{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——LightGBM\n",
    "原始特征+tfidf特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，对LightGBM的超参数进行调优。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.081393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.075886</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.231403</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.199730</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.011987</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.011668</td>\n",
       "      <td>0.105971</td>\n",
       "      <td>0.021681</td>\n",
       "      <td>0.080435</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008244</td>\n",
       "      <td>0.022456</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.124622</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.145988</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 95 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0   1      0.081393           0.0           0.0      0.000000      0.000000   \n",
       "1   2      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "2   3      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "3   4      0.011987           0.0           0.0      0.011668      0.105971   \n",
       "4   5      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  ...  feat_85_tfidf  \\\n",
       "0      0.000000      0.000000      0.000000           0.0  ...       0.075886   \n",
       "1      0.000000      0.000000      0.231403           0.0  ...       0.000000   \n",
       "2      0.000000      0.000000      0.199730           0.0  ...       0.000000   \n",
       "3      0.021681      0.080435      0.000000           0.0  ...       0.000000   \n",
       "4      0.000000      0.000000      0.000000           0.0  ...       0.124622   \n",
       "\n",
       "   feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  feat_90_tfidf  \\\n",
       "0       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "1       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "2       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "3       0.008244       0.022456            0.0            0.0       0.000000   \n",
       "4       0.000000       0.000000            0.0            0.0       0.145988   \n",
       "\n",
       "   feat_91_tfidf  feat_92_tfidf  feat_93_tfidf   target  \n",
       "0            0.0            0.0            0.0  Class_1  \n",
       "1            0.0            0.0            0.0  Class_1  \n",
       "2            0.0            0.0            0.0  Class_1  \n",
       "3            0.0            0.0            0.0  Class_1  \n",
       "4            0.0            0.0            0.0  Class_1  \n",
       "\n",
       "[5 rows x 95 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# 这里使用tf_idf特征\n",
    "\n",
    "train = pd.read_csv(\"./Otto_FE_train_tfidf.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将类别字符串变成数字，LightGBM不支持字符串格式的特征输入/标签输入\n",
    "y_train = train['target'] #形式为Class_x\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s) - 1)#将类别的形式由Class_x变为0-8之间的整数\n",
    "\n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM超参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM的主要的超参包括：\n",
    "1. 树的数目n_estimators 和 学习率 learning_rate\n",
    "2. 树的最大深度max_depth 和 树的最大叶子节点数目num_leaves（注意：XGBoost只有max_depth，LightGBM采用叶子优先的方式生成树，num_leaves很重要，设置成比 2^max_depth 小）\n",
    "3. 叶子结点的最小样本数:min_data_in_leaf(min_data, min_child_samples)\n",
    "4. 每棵树的列采样比例：feature_fraction/colsample_bytree\n",
    "5. 每棵树的行采样比例：bagging_fraction （需同时设置bagging_freq=1）/subsample\n",
    "6. 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)\n",
    "\n",
    "7. 两个非模型复杂度参数，但会影响模型速度和精度。可根据特征取值范围和样本数目修改这两个参数\n",
    "1）特征的最大bin数目max_bin：默认255；\n",
    "2）用来建立直方图的样本数目subsample_for_bin：默认200000。\n",
    "\n",
    "对n_estimators，用LightGBM内嵌的cv函数调优，因为同XGBoost一样，LightGBM学习的过程内嵌了cv，速度极快。\n",
    "其他参数用GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 相同的交叉验证分组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#直接调用lightgbm内嵌的交叉验证(cv)，可对连续的n_estimators参数进行快速交叉验证\n",
    "#而GridSearchCV只能对有限个参数进行交叉验证，且速度相对较慢\n",
    "def get_n_estimators(params , X_train , y_train , early_stopping_rounds=10):\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "     \n",
    "    lgbmtrain = lgbm.Dataset(X_train , y_train )\n",
    "     \n",
    "    #num_boost_round为弱分类器数目，下面的代码参数里因为已经设置了early_stopping_rounds\n",
    "    #即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    cv_result = lgbm.cv(lgbm_params , lgbmtrain , num_boost_round=MAX_ROUNDS , nfold=3,  metrics='multi_logloss' , early_stopping_rounds=early_stopping_rounds,seed=3)\n",
    "     \n",
    "    print('best n_estimators:' , len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score:' , cv_result['multi_logloss-mean'][-1])\n",
    "     \n",
    "    return len(cv_result['multi_logloss-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 394\n",
      "best cv score: 0.4878865392203175\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'num_leaves': 60,\n",
    "          'max_depth': 6,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "\n",
    "n_estimators_1 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. num_leaves & max_depth=7\n",
    "num_leaves建议70-80，搜索区间50-80,值越大模型越复杂，越容易过拟合\n",
    "相应的扩大max_depth=7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  7.6min remaining:  3.8min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed: 11.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=31,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.7,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'num_leaves': range(50, 90, 10)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "num_leaves_s = range(50,90,10) #50,60,70,80\n",
    "tuned_parameters = dict( num_leaves = num_leaves_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4, param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48720982947663044\n",
      "{'num_leaves': 70}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEHCAYAAAB4POvAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxV5b3v8c83IxACiInKEE3AhB4cCpoi1jqBoLY92EFbHFqP9R5rK07g7dHbe/s6V29ft9firMdW26LtaR2rPdS2Cg44YFECFRARDHMQITgwQ6bf/WOt4GKzIRvJYmUnv/frtV8761nPs/bvcUl+eZ619npkZjjnnHNxykk6AOecc52fJxvnnHOx82TjnHMudp5snHPOxc6TjXPOudjlJR1AR1RSUmLl5eVJh+Gcc1llzpw5G8ysNN0+TzZplJeXU1NTk3QYzjmXVSSt3Ns+n0ZzzjkXO082zjnnYufJxjnnXOw82TjnnIudJxvnnHOx82TjnHMudp5snHPOxc6/Z9OONmzZyd8WrCUnR+TliByJ3JxPX61lebnhe04OOTmQl5NDbg5py3JzcsiVyM1V8N56vDRlOQJJSf9ncM65PXiyaUd1H2/nf/3XwkRjiCajvByRk5OSoKKJL00C3K39rqQYqRu+52rPsta60QSb2j439aUMy1JiSpvEI4l6V8KWyMmBnoV5noidS1CsyUbSOcBdQC7wKzP72V7qnQ88AXzBzGok5QO/Ak4IY/ytmf1fSUOAxyJNBwE/MbM7w+NcDUwAmoC/mNmPwvKbgMuBZuAaM3uu/XsLx/TvRc3/PIvmFtvt1dRitFj6sqbm8L3FaEndF5ZF35ttz7LW4zSb0dzSQnMLu96D4+xe1tzSQrO1blvasqaWFnY2hfFG4tzVBzOad31mJO6W3ctaOsjafF85rh/3XXxC0mE412XFlmwk5QL3AWOAOmC2pKlm9k5KvWLgGuCNSPEFQKGZHSepB/COpEfMbDEwLHL8NcDT4faZwHnA8Wa2U9JhYflQYDxwDNAfeF5SlZk1t3ef83NzKOlZ2N6HzWpm+06gzWkSblPL7kktXcJNTXS7J9zdXwvf38Qf59Zx8dINfHFwSdL/SZzrkuIc2YwAas1sGYCkRwmSwTsp9W4BbgVuiJQZUCQpD+gONACbUtqNBpaaWeuzeH4A/MzMdgKY2fqw/Dzg0bB8uaTaMLa/H3gXXVsUTn3l5SYXw47GZmbWbuC2aUs4+cpDfTrNuQTEeTfaAGB1ZLsuLNtF0nCgzMyeSWn7JLAVWAusAiab2UcpdcYDj0S2q4BTJb0h6WVJX8g0jjCWKyTVSKqpr6/PqIMuO3TLz+Xq0UczZ+XHzFji59a5JMSZbNL9+bhrBl9SDnAHMClNvREE11f6AxXAJEmDIm0LgHEE13la5QGHACOB/w48ruBP2H3GsavA7AEzqzaz6tLStE/IdlnsghPLKOvbndumLcasg1xIcq4LiTPZ1AFlke2BwPuR7WLgWGCGpBUESWKqpGrgIuBZM2sMp8NmAtWRtucCc81sXcrnPWWBN4EWoCSDOFwXUJCXw7Wjq3h7zSaeW/hB0uE41+XEmWxmA5WSKsKRyHhgautOM9toZiVmVm5m5cAsYJyZ1RBMnY1SoIggEb0bOfaF7D6FBvAnYBSApCqgANgQfuZ4SYWSKoBK4M32767r6L4+fACDS4u4ffoSmjvKbXLOdRGxJRszayK4Dfk5YBHwuJktlHSzpHFtNL8P6Am8TZC0ppjZfIDw7rQxwFMpbX4DDJL0NvAocGk4ylkIPE5wY8KzwFVx3InmOr7cHHH9mCqWrNvCM/N9cOvcwSSfv95TdXW1+UqdnVNLi/Hlu19lR2Mz0yeeTn6uP7HJufYiaY6ZVafb5//SXJeSkyMmjR3Cig+38dTcuqTDca7L8GTjupyz/ukwPl/Wh7tfqGVnk8+oOncweLJxXY4kbhhbxZpPtvPom6vbbuCcO2CebFyX9KWjSxhR0Zd7X6ple4OPbpyLmycb1yUFo5sh1G/eye9mrUg6HOc6PU82rssaUdGX06pKuX/GUjbvaEw6HOc6NU82rkubNKaKj7c1MmXmiqRDca5T82TjurTPl/Vh7NDDefCVZXyyrSHpcJzrtDzZuC5v4tgqtjQ08eCry5IOxblOy5ON6/I+d0Qvvnp8f6bMXMGGLTuTDse5TsmTjXPA9WdVsqOxmftnLE06FOc6JU82zgGDSnvyzRMG8rtZK/lg446kw3Gu0/Fk41zomtGVmBn3vPhe0qE41+l4snEuVNa3B9/+QhmPzV7N6o+2JR2Oc52KJxvnIq4eVUlujrjrBR/dONeePNk4F3F4r258Z+RRPDW3jtr1W5IOx7lOI9ZkI+kcSYsl1Uq6cR/1zpdkkqrD7XxJD0taIGmRpJvC8iGS3oq8Nkm6Ltz375LWRPZ9OSwvl7Q9Uv6LOPvsst+VZwymW34udz6/JOlQnOs08uI6sKRcguWdxwB1wGxJU83snZR6xcA1wBuR4guAQjM7LlwG+h1Jj5jZYmBY5PhrgKcj7e4ws8lpwllqZsPaq2+ucyvpWcj3Tqng3pdquerMTfxTv15Jh+Rc1otzZDMCqDWzZWbWADwKnJem3i3ArUD0flMDiiTlAd2BBmBTSrvRBElkZbtH7rq8fz11EMXd8rhtmo9unGsPcSabAUB0Zaq6sGwXScOBMjN7JqXtk8BWYC2wCphsZh+l1BkPPJJSNkHSfEm/kXRIpLxC0j8kvSzp1HTBSrpCUo2kmvr6+ow66Dqv3j3yueLUQTy/aB1vrf4k6XCcy3pxJhulKbNdO6Uc4A5gUpp6I4BmoD9QAUySNCjStgAYBzwRaXM/MJhgmm0tcFtYvhY40syGAxOBP0jaY17EzB4ws2ozqy4tLc24k67zuuxLFfQtKuC2aYuTDsW5rBdnsqkDyiLbA4H3I9vFwLHADEkrgJHA1PAmgYuAZ82s0czWAzOB6kjbc4G5ZrautcDM1plZs5m1AA8SJCzMbKeZfRj+PAdYClS1a09dp9SzMI8fnD6YV9/bwBvLPkw6HOeyWpzJZjZQKakiHImMB6a27jSzjWZWYmblZlYOzALGmVkNwdTZKAWKCBLRu5FjX0jKFJqkfpHNrwNvh+Wl4c0EhKOjSsAf7+sycsnIozisuJDbpi3BzNpu4JxLK7ZkY2ZNwATgOWAR8LiZLZR0s6RxbTS/D+hJkDBmA1PMbD5AeHfaGOCplDa3hrdKzwfOBK4Py08D5kuaR3At6Mo013+cS6t7QS4TRh3Nmys+4tX3NiQdjnNZS/7X2p6qq6utpqYm6TBcB7GzqZlRk1+mpGcBf7rqFKR0lyOdc5LmmFl1un3+BAHn2lCYl8u1oyuZV7eR6e+sa7uBc24Pnmycy8A3ThhARUkRt09fQkuLzwY4t7882TiXgbzcHK47q5J3P9jMXxasTToc57KOJxvnMvTPx/dnyOHF3DF9CU3NLUmH41xW8WTjXIZycsT1Y6pYtmErT/9jTdLhOJdVPNk4tx/OPuZwjhvQm7teeI+GJh/dOJcpTzbO7QdJTBpbRd3H23msZnXbDZxzgCcb5/bb6VWlVB91CPe++B47GpuTDse5rODJxrn9FIxuhrBu007+c5avcOFcJjzZOPcZnDz4UL50dAn3z1jK1p1NSYfjXIfnyca5z2ji2Co+3NrAQ6+vSDoU5zo8TzbOfUYnHHkIoz93GL98eSkbtzcmHY5zHZonG+cOwMSxVWza0cSvX/VVK5zbF082zh2AY/r35ivH9ePXry3nwy07kw7HuQ7Lk41zB+j6MZVsb2zml6/46Ma5vfFk49wBOvqwYr42fAAPv76C9Zt2JB2Ocx1SrMlG0jmSFkuqlXTjPuqdL8kkVYfb+ZIeDlfeXCTpprB8iKS3Iq9Nkq4L9/27pDWRfV+OHP+mMIbFks6Os8+ua7p2dCXNLca9L9UmHYpzHVJeXAeWlEuwvPMYoA6YLWmqmb2TUq8YuAZ4I1J8AVBoZseFy0C/I+kRM1sMDIscfw3wdKTdHWY2OeX4Q4HxwDFAf+B5SVVm5l/9du3mqEOLuKC6jEfeXMUVpw1i4CE9kg7JuQ4lzpHNCKDWzJaZWQPwKHBemnq3ALcC0fkHA4ok5QHdgQZgU0q70cBSM2vrK9znAY+a2U4zWw7UhrE5166uGX00krjnBR/dOJcqzmQzAIg+qbAuLNtF0nCgzMyeSWn7JLAVWAusAiab2UcpdcYDj6SUTZA0X9JvJB2SaRxhLFdIqpFUU19f33bvnEvRr3d3Lj7pSJ6cW8fyDVuTDse5DiXOZKM0ZbvW05WUA9wBTEpTbwTQTDDtVQFMkjQo0rYAGAc8EWlzPzCYYJptLXBbJnHsKjB7wMyqzay6tLR0H91ybu9+cMZgCnJzuPP5JUmH4lyHEmeyqQPKItsDgfcj28XAscAMSSuAkcDU8CaBi4BnzazRzNYDM4HqSNtzgblmtq61wMzWmVmzmbUAD/LpVFlbcTjXbg4r7salXyxn6rz3WfzB5qTDca7DiDPZzAYqJVWEI5HxwNTWnWa20cxKzKzczMqBWcA4M6shmDobpUARQSJ6N3LsC0mZQpPUL7L5deDt8OepwHhJhZIqgErgzfbsqHNRV54+iJ4Fedwx3Uc3zrWKLdmYWRMwAXgOWAQ8bmYLJd0saVwbze8DehIkjNnAFDObDxDenTYGeCqlza3hrdLzgTOB68M4FgKPA+8AzwJX+Z1oLk59ehRw+akVPLvwAxbUbUw6HOc6BJntcfmiy6uurraampqkw3BZbPOORk699SWGlfXhocv85kfXNUiaY2bV6fb5EwSci0Fxt3yuPH0wMxbXU7Mi9UZK57oeTzbOxeS7Jx9FSc9CJk9bjM8guK7Ok41zMelRkMdVZw5m1rKPeH3ph0mH41yiPNk4F6OLTjqS/r27+ejGdXmebJyLUWFeLlePruQfqz7hxXfXJx2Oc4nxZONczM4/cSBH9u3BbdOW0NLioxvXNXmycS5m+bk5XHdWJe+s3cSzCz9IOhznEuHJxrmD4LxhAzj6sJ7cPn0JzT66cV2QJxvnDoLcHDFxTBW167fwX2+tSToc5w46TzbOHSTnHHMEQ/v14s7n36OxuSXpcJw7qDzZOHeQ5OSIG86uYtVH23hyTl3S4Th3UHmyce4gOnPIYQw/sg93v/AeOxr9ebCu6/Bk49xBJIkbxg5h7cYdPPLmqqTDce6g8WTj3EF2ytElnDzoUO57aSnbGpqSDse5g8KTjXMJmDS2ig1bdvLw6yuTDsW5g8KTjXMJqC7vyxlDSvnFy0vZtKMx6XCci12syUbSOZIWS6qVdOM+6p0vySRVh9v5kh4OV95cJOmmsHyIpLcir02Srks51g3hsUrC7TMkbYy0+UmcfXYuU5PGDGHj9kZ+89rypENxLnZ5cR1YUi7B8s5jgDpgtqSpZvZOSr1i4BrgjUjxBUChmR0XLgP9jqRHzGwxMCxy/DXA05FjlYWfl3rl9VUz+2q7dtC5A3TcwN6cc8wR/OrV5Vx6cjmHFBUkHZJzsYlzZDMCqDWzZWbWADwKnJem3i3ArcCOSJkBRZLygO5AA7Appd1oYKmZRSe97wB+FLZ3rsO7fkwVWxua+OUry5IOxblYtZlsJA2WVBj+fIakayT1yeDYA4DVke26sCx67OFAmZk9k9L2SWArsJZglDLZzFLX1h0PPBI51jhgjZnNSxPLyZLmSfqbpGPSBSvpCkk1kmrq6+sz6J5zB27IEcWM+3x/Hnp9Oes372i7gXNZKpORzR+BZklHA78GKoA/ZNBOacp2jTgk5RCMRCalqTcCaAb6h583SdKgSNsCYBzwRLjdA/gxkO56zFzgKDP7PHAP8Kd0wZrZA2ZWbWbVpaWlbffOuXZy3VlVNDYb989YmnQozsUmk2TTYmZNwNeBO83seqBfBu3qgLLI9kDg/ch2MXAsMEPSCmAkMDW8SeAi4FkzazSz9cBMoDrS9lxgrpmtC7cHEySleeGxBgJzJR1hZpvMbAuAmf0VyG+9ecC5jqCipIjzTxjI72et4v1PticdjnOxyCTZNEq6ELgUaJ3uys+g3WygUlJFOBIZD0xt3WlmG82sxMzKzawcmAWMM7MagqmzUQoUESSidyPHvpDIFJqZLTCzwyLHqgNOMLMPJB0hSQCSRoR99gXhXYdy9eijMYx7XqxNOhTnYpFJsrkMOBn4qZktl1QB/GdbjcLR0ATgOWAR8LiZLZR0c3h9ZV/uA3oCbxMkrSlmNh92TZmNAZ7KIHaA84G3Jc0D7gbGmy8G7zqYgYf04KIRR/JEzWpWfrg16XCca3fan9+7kg4huKA/P76QklddXW01NTVJh+G6mPWbdnDqrS/xleP6cfu3hyUdjnP7TdIcM6tOty+Tu9FmSOolqS8wD5gi6fb2DtK5ru6wXt249IvlPP3WGt5btznpcJxrV5lMo/U2s03ANwims04Ezoo3LOe6pitPH0yP/FzufP69pENxrl1lkmzyJPUDvsWnNwg452LQt6iAy79UwV8WrOXtNRuTDse5dpNJsrmZ4CL/UjObHX7fxf/sci4ml586iF7d8rhj+pKkQ3Gu3bSZbMzsCTM73sx+EG4vM7Nvxh+ac11T7+75fP/0wbzw7nrmrvo46XCcaxeZ3CAwUNLTktZLWifpj5IGHozgnOuq/uWL5RxaVMBt0xYnHYpz7SKTabQpBF/G7E/wbLM/h2XOuZgUFebxgzMGM7P2Q15fuiHpcJw7YJkkm1Izm2JmTeHrIcAfHuZczC4ZeRSH9yrk9mlL8O8hu2yXSbLZIOkSSbnh6xL8cS/Oxa5bfi5Xj6qkZuXHvLzEn0TuslsmyeZ7BLc9f0DwyP/zCR5h45yL2beqyxh4SHdu89GNy3KZ3I22yszGmVlp+LDLrxF8wdM5F7OCvByuHV3JgjUbeW7hurYbONdBfdaVOie2axTOub36+vABDCot4vbpi2lu8dGNy06fNdmkWxjNOReDvNwcrj+riiXrtvDM/PfbbuBcB/RZk43/eeXcQfSV4/rxuSOKuWP6EpqaW5IOx7n9ttdkI2mzpE1pXpsJvnPjnDtIcnLExDFVrPhwG0/NXZN0OM7tt70mGzMrNrNeaV7FZpaXycElnSNpsaRaSTfuo975kixcEhpJ+ZIelrRA0iJJN4XlQyS9FXltknRdyrFuCI9VEm5L0t1hDPMlnZBJ7M51NGOGHs7nB/bmrhfeY2dTc9LhOLdfPus0Wpsk5RKsuHkuMBS4UNLQNPWKgWuANyLFFwCFZnYccCLwfUnlZrbYzIaZ2bCwfBvwdORYZQSreK6KHOtcoDJ8XQHc3369dO7gkcSksUNY88l2Hpu9OulwnNsvsSUbYARQGz64swF4FDgvTb1bgFuBHZEyA4ok5QHdgQZgU0q70QRPol4ZKbsD+BG7X1M6D/itBWYBfcIlE5zLOqdWljCivC/3vFjL9gYf3bjsEWeyGQBE//yqC8t2kTScYJnp1HVyngS2EnyJdBUw2cw+SqkzHngkcqxxwBozm7e/cTiXLYLRTRX1m3fyn7NWtt3AuQ4izmST7vboXSMOSTkEI5FJaeqNAJoJbkSoACaF6+i0ti0AxgFPhNs9gB8DP9nfOCLHvEJSjaSa+np/NIjruE4adCinVpZw/8tL2bKzKelwnMtIJksMpLsrbXW47MCgfTStA8oi2wOB6JcEioFjgRmSVgAjganhTQIXAc+aWaOZrQdmAtWRtucCc82s9SvVgwmS0rzwWAOBuZKOyCAOAMzsATOrNrPq0lJ/zqjr2CaNHcJHWxuY8trypENxLiOZjGxuB/47wdTTQOAG4EGCazC/2Ue72UClpIpwJDKeYKkCAMxso5mVmFm5mZUDs4BxZlZDMHU2KryTrIggEb0bOfaFRKbQzGxB+Cid1mPVASeY2QfhZ343PNZIYKOZrc2g3851WMPK+jBm6OE88OoyNm5rTDoc59qUSbI5x8x+aWabzWyTmT0AfNnMHgMO2VsjM2sCJhAsKb0IeNzMFkq6Oby+si/3AT2BtwmS1hQzmw+7pszGAE9lEDvAX4FlQC1Bkvxhhu2c69Amjqli844mHnh1adKhONemTL4v0yLpWwQX7SF46nOrfT5JwMz+SvDLPlqW7roKZnZG5OctBLc/p6u3DTi0jc8tj/xswFX7qu9cNvqnfr346vH9mDJzBZedUkFJz8KkQ3JurzIZ2VwMfAdYH76+A1wiqTvByMU5l5DrzqpiR2Mzv5jhoxvXsWWyxMAyM/vn8PpKSfhzrZltN7PXDkaQzrn0jj6sJ984YSC/nbWSDzbuaLuBcwnJ5G60geGdZ+slrZP0R0kDD0Zwzrm2XTu6kpYW496X3ks6FOf2KpNptCkEd3T1J7gj7c9hmXOuAyjr24Nvf6GMx2avZvVH25IOx7m0Mkk2pWY2xcyawtdDgH8RxbkO5OpRlUjirhd8dOM6pkySzQZJl0jKDV+XAB/GHZhzLnNH9O7Gd0YexVNz61havyXpcJzbQybJ5nvAt4APCJ5Vdj5wWZxBOef23w/OGEy3/FzufN5HN67jyeRutFVmNs7MSsNv6X8N+MZBiM05tx9KehZy2Snl/Hne+yxam/qQdOeS9VkfxDmxXaNwzrWLK04dTHG3PG6fviTpUJzbzWdNNumepOycS1jvHvn866mDmP7OOuat/iTpcJzb5bMmm30+psY5l5zvfamCQ3rkc5uPblwHstdks5elBTZJ2kzwnRvnXAfUszCPH5wxmFeW1PPm8tQ1B51Lxl6TjZkVm1mvNK9iM8vkAZ7OuYR8Z2Q5pcWFTH5uMcGzaJ1LVpwrdTrnEtK9IJcJZx7Nmys+4rXaDUmH45wnG+c6q/EjyhjQp7uPblyH4MnGuU6qMC+Xa0Yfzby6jTy/aH3S4bguLtZkI+kcSYsl1Uq6cR/1zpdkkqrD7XxJD0taIGmRpJvC8iGS3oq8Nkm6Ltx3i6T5Yfk0Sf3D8jMkbYy0Sbt4m3Od0TdOGEj5oT24bdpiWlp8dOOSE1uykZRLsLzzucBQ4EJJQ9PUKwauAd6IFF8AFJrZccCJwPcllZvZYjMbZmbDwvJtwNNhm5+b2fHhvmeAaFJ5tbWdmd3czl11rsPKz83h+jFVvPvBZv6yYG3S4bguLM6RzQigNlx8rQF4FDgvTb1bgFuB6MpPBhRJygO6Aw1A6vM3RgNLzWwlgJlF9xfh3wVyDoCvHt+fqsN7csfzS2hqbkk6HNdFxZlsBgCrI9t1YdkukoYDZWb2TErbJ4GtBA/+XAVMNrPULwyMBx5JOd5PJa0mWMo6OrI5WdI8SX+TdEy6YCVdIalGUk19fX1mPXQuC+TmiIljqlhWv5U/vfV+0uG4LirOZJPukTa7RhuScoA7gElp6o0Amgm+PFoBTJI0KNK2ABgHPLHbwc1+bGZlwO+BCWHxXOAoM/s8cA/wp3TBmtkDZlZtZtWlpb5cj+tczj7mCI4d0Iu7XlhCQ5OPbtzBF2eyqQPKItsDgeifVcXAscAMSSuAkcDU8CaBi4BnzazRzNYDM4HqSNtzgblmtm4vn/0H4JsQTK+Z2Zbw578C+ZJKDrRzzmUTSUwaO4TVH23n8ZrVbTdwrp3FmWxmA5WSKsKRyHiC5aUBMLONZlZiZuVmVg7MAsaZWQ3B1NkoBYoIEtG7kWNfyJ5TaJWRzXGt9SUdIUnhzyMI+uyLv7ku54yqUk486hDuefE9djQ2Jx2O62JiSzZm1kQwlfUcsAh43MwWSrpZ0rg2mt8H9ATeJkhaU8xsPoCkHsAY4KmUNj+T9Lak+cBY4Nqw/HzgbUnzgLuB8ebfcHNdUDC6qWLdpp38/o1VSYfjuhj57909VVdXW01NTdJhOBeLi381i3fXbuaVH51JUaE/5tC1H0lzzKw63T5/goBzXczEMUP4cGsDD72+IulQXBfiyca5LubEow5h1OcO45cvL2Xj9sakw3FdhCcb57qgiWOq2LSjiV+/uizpUFwX4cnGuS7o2AG9+fJxR/Dr15bz0daGpMNxXYAnG+e6qOvPqmJbYzO/fHlp0qG4LsCTjXNdVOXhxXx92AAe/vsK1m/a0WZ95w6EJxvnurBrz6qksdm476XapENxnZwnG+e6sKMOLeJb1QP5w5urqPt4W9LhuE7Mk41zXdyEUZUIce+LPrpx8fFk41wXN6BPdy466UiemFPH8g1bkw7HdVKebJxz/PDMweTnirueX5J0KK6T8mTjnOOw4m5c+sVy/mve+yxZtznpcFwn5MnGOQfAlacNpqggj9un+ejGtT9PNs45AA4pKuDyL1Xw7MIPWFC3MelwXCfjycY5t8vlp1bQu3s+t09fnHQorpPxZOOc26VXt3yuPH0wLy2uZ87Kj5IOx3UisSYbSedIWiypVtKN+6h3viSTVB1u50t6WNICSYsk3RSWD5H0VuS1SdJ14b5bJM0Py6dJ6h+WS9LdYQzzJZ0QZ5+dy3aXfvEoSnoWMPk5v3bj2k9syUZSLsHyzucCQ4ELJQ1NU68YuAZ4I1J8AVBoZscBJwLfl1RuZovNbJiZDQvLtwFPh21+bmbHh/ueAX4Slp8LVIavK4D727mrznUqPQry+OEZR/P3ZR/yeu2GpMNxnUScI5sRQK2ZLTOzBuBR4Lw09W4BbgWiTwI0oEhSHtAdaAA2pbQbDSw1s5UAZhbdXxQeg/Azf2uBWUAfSf0OrGvOdW4XnXQk/Xp3Y/K0xfjS8a49xJlsBgCrI9t1YdkukoYDZWb2TErbJ4GtwFpgFTDZzFInkMcDj6Qc76eSVgMX8+nIps04wrZXSKqRVFNfX59B95zrvLrl53L1qErmrvqElxavTzoc1wnEmWyUpmzXn0iScoA7gElp6o0AmoH+QAUwSdKgSNsCYBzwxG4HN/uxmZUBvwcmZBJHpO0DZlZtZtWlpaX76pdzXcIF1QM5sm8Pbpu2hJYWH924AxNnsqkDyiLbA4H3I9vFwLHADEkrgJHA1PAmgYuAZ82s0czWAzOB6kjbc4G5ZrZuL5/9B+CbGcbhnEsjPzeHa0dXsvD9TTy38IOkw3FZLs5kMxuolFQRjkTGA5R3t0oAABBQSURBVFNbd5rZRjMrMbNyMysHZgHjzKyGYOpsVHgnWRFBIno3cuwL2XMKrTKyOS5Sfyrw3fBYI4GNZra2XXvqXCf1teEDGFxaxG3Tl9Dsoxt3AGJLNmbWRDCV9RywCHjczBZKulnSuDaa3wf0BN4mSFpTzGw+gKQewBjgqZQ2P5P0tqT5wFjg2rD8r8AyoBZ4EPjhAXfOuS4iN0dMHDOE2vVbmDpvTdLhuCwmv9NkT9XV1VZTU5N0GM51CC0txlfueY1tDU08P/F08nP9u+AuPUlzzKw63T7/v8Y5t085OeKGsVWs/HAbT86pSzocl6U82Tjn2jTqc4cxrKwPd7/wHjsam5MOx2UhTzbOuTZJ4oaxQ1i7cQePvrkq6XBcFvJk45zLyClHH8rIQX2596WlbGtoSjocl2U82TjnMiKJSWOHsGHLTn7795VJh+OyjCcb51zGvlDel9OrSvnFy0vZvKMx6XBcFvFk45zbL5PGVvHJtkZ+89qKpENxWcSTjXNuvxw/sA9nH3M4v3p1GZ9sa0g6HJclPNk45/bb9WOq2NLQxC9fWZZ0KC5LeLJxzu23zx3Ri38+vj8PzVxB/eadSYfjsoAnG+fcZ3LdWZU0NLfwHzNqkw7FZQFPNs65z2RQaU++ecIAfj9rFe9/sj3pcFwH58nGOfeZXT2qEsO49yUf3bh982TjnPvMyvr24MIRR/L47NWs+nBb0uG4DsyTjXPugFx15tHk5og7X1iSdCiuA/Nk45w7IIf36sZ3Tz6KP/1jDbXrNycdjuugYk02ks6RtFhSraQb91HvfEkmqTrczpf0sKQFkhZJuiksHyLprchrk6Trwn0/l/SupPmSnpbUJywvl7Q90uYXcfbZua7oytMH0z0/lzumv5d0KK6Dii3ZSMolWN75XGAocKGkoWnqFQPXAG9Eii8ACs3sOOBE4PuSys1ssZkNM7NhYfk24OmwzXTgWDM7HlgC3BQ53tLWdmZ2Zfv21Dl3aM9CvvelCv6yYC0L39+YdDiuA4pzZDMCqDWzZWbWADwKnJem3i3ArcCOSJkBRZLygO5AA7Appd1ogiSyEsDMpplZ63PPZwED260nzrk2/bdTB9GrWx53TPdrN25PcSabAcDqyHZdWLaLpOFAmZk9k9L2SWArsBZYBUw2s49S6owHHtnLZ38P+Ftku0LSPyS9LOnUdA0kXSGpRlJNfX39vvrlnEujd/d8rjhtEM8vWs8/Vn2cdDiug4kz2ShNme3aKeUAdwCT0tQbATQD/YEKYJKkQZG2BcA44Ik9PlT6MdAE/D4sWgscaWbDgYnAHyT12iMwswfMrNrMqktLSzProXNuN5edUkHfogJum+ajG7e7OJNNHVAW2R4IvB/ZLgaOBWZIWgGMBKaGNwlcBDxrZo1mth6YCVRH2p4LzDWzddEPlHQp8FXgYjMzADPbaWYfhj/PAZYCVe3WS+fcLkWFefzwjMG8VruBvy/9MOlwXAcSZ7KZDVRKqghHIuOBqa07zWyjmZWYWbmZlRNcZxlnZjUEU2ejFCgiSETvRo59ISlTaJLOAf4tPMa2SHlpeLMC4eioEvBH1ToXk0tGHsXhvQq5ffpiwr/5nIsv2YQX6ycAzwGLgMfNbKGkmyWNa6P5fUBP4G2CpDXFzOYDSOoBjAGeSmlzL8FoaXrKLc6nAfMlzSO4FnRlmus/zrl20i0/lwmjKpm94mNeXuLXP11A/pfHnqqrq62mpibpMJzLWg1NLZw5eQZ9iwqYOuEUpHSXcF1nI2mOmVWn2+dPEHDOtbuCvByuPauSBWs2Mu2ddW03cJ2eJxvnXCy+MXwAg0qKuH3aEppbfAalq/Nk45yLRV5uDteNqWLxus08M//9thu4Ts2TjXMuNl89rh9DDi/mzuffo6m5JelwXII82TjnYpOTIyaOrWL5hq08NXdN0uG4BHmycc7FauzQwzl+YG/ueuE9djY1Jx2OS4gnG+dcrCQxaewQ1nyyncdnr267geuUPNk452J3WmUJXyg/hHterGVHo49uuqK8pANwznV+krhh7BC+/cAsfvf3lfzraYPabuTaTUuLsb2xma0NTWzbGb43NLN1Z8p7QxODSoo459h+7R6DJxvn3EFx0qBDObWyhPtfXsqFJx1Jz0L/9ZNOY3NLJCE0sXXnPpJEtDyyP3gFbbeFZZn6ynH9PNk457LbpLFD+Np9M3lo5nImjKpMOpwDYmbsbGrZ7Rd/6y/3Xe8NzWzbmfKeZn80cTTsxy3ihXk5FBXm0aMgl6KCPHoUBu+HFhV8Wl6YR/f8XIoKc+lRkPfpe6R+a70eBbkU5sVzdcWTjXPuoBlW1oez/ulwfvnKMr4zspzePfIPyuc2t9iuv/DTTR21NbWUbv+2hib258EIRQW59CjMC97DX/p9uuczoE+38Jf/nvv3mhwKc+mRn0tebvZcdvdk45w7qCaOqeLLd7/Kg68u44azh+y2z8xo2G0aKZwSSjcq2DU6aHuKaUdj5qOFvBxRVLjnL/9+vbvt9su/R2pS2Eey6JaXS05O134YqScb59xBNbR/L75yfD8efHUZr7xXv8dIomk/hgvd8nPSTgeVFhemlKcbKUSSQyRJFMQ0jdTVebJxzh10Pzp7CJu2N5KXI8r69thjNJB6DSJ6TaH1vUdBHrldfLSQTTzZOOcOuqMOLeJ3l5+UdBjuIIp1vCjpHEmLJdVKunEf9c6XZJKqw+18SQ9LWiBpkaSbwvIh4Sqcra9Nkq4L9/1c0ruS5kt6WlKfyPFvCmNYLOnsOPvsnHNuT7ElG0m5BMs7nwsMBS6UNDRNvWLgGuCNSPEFQKGZHQecCHxfUrmZLTazYWY2LCzfBjwdtpkOHGtmxwNLgNYENRQYDxwDnAP8Rxibc865gyTOkc0IoNbMlplZA/AocF6aercAtwI7ImUGFEnKA7oDDcCmlHajgaVmthLAzKaZWVO4bxYwMPz5POBRM9tpZsuB2jA255xzB0mcyWYAEH3qXl1Ytouk4UCZmT2T0vZJYCuwFlgFTDazj1LqjAce2ctnfw/4W6ZxhLFcIalGUk19ff1eO+Wcc27/xZls0t0msuueRkk5wB3ApDT1RgDNQH+gApgkaVCkbQEwDnhijw+Vfgw0Ab/PJI5dBWYPmFm1mVWXlpburU/OOec+gzjvRqsDyiLbA4Ho2rDFwLHADEkARwBTJY0DLgKeNbNGYL2kmUA1sCxsey4w18zWRT9Q0qXAV4HRZtaaUNqKwznnXMziHNnMBiolVYQjkfHA1NadZrbRzErMrNzMygmus4wzsxqCqbNRChQBI4F3I8e+kJQpNEnnAP8WHmNbZNdUYLykQkkVQCXwZnt31jnn3N7FlmzCi/UTgOeARcDjZrZQ0s3h6GVf7gN6Am8TJK0pZjYfQFIPYAzwVEqbewlGS9PD26J/EcaxEHgceAd4FrjKzHxBDeecO4j06WyTayWpHlh5AIcoATa0UzhJ6iz9AO9LR9RZ+gHel1ZHmVnai96ebGIgqcbMqpOO40B1ln6A96Uj6iz9AO9LJvyJc84552LnycY551zsPNnE44GkA2gnnaUf4H3piDpLP8D70ia/ZuOccy52PrJxzjkXO082zjnnYufJ5gBJWhGuu/OWpJqwrK+k6ZLeC98PSTrOTOylL/8uaU1kDaEvJx1nWyT1kfRkuL7RIkknZ/E5SdeXbDwnadeiysbzso++ZON5uV7SQklvS3pEUrfwqS9vhOfksfAJMAf+WX7N5sBIWgFUm9mGSNmtwEdm9jMFi8YdYmb/llSMmdpLX/4d2GJmk5OKa39Jehh41cx+Ff5D6QH8D7LznKTry3Vk2TmJCteTWgOcBFxFFp6XVil9uYwsOi+SBgCvAUPNbLukx4G/Al8GnjKzR8Mnscwzs/sP9PN8ZBOP84CHw58fBr6WYCxdiqRewGnArwHMrMHMPiELz8k++pLtomtRZd15SbHbulpZKA/oHq4d1oNgWZdRBMu8QDueE082B86AaZLmSLoiLDvczNYChO+HJRbd/knXF4AJCpbb/k0WTHMMAuqBKZL+IelX4cNcs/Gc7K0vkF3nJFV0LapsPC9RqetqZc15MbM1wGSCBx+vBTYCc4BPIgtRpl3/67PwZHPgTjGzEwiWPbhK0mlJB3QA0vXlfmAwMIzgf8jbEowvE3nACcD9ZjacYBG+G5MN6TPbW1+y7Zzson2sRZVt0vQlq85LmAzPI1gzrD9QRPBvP1W7XGvxZHOAzOz98H098DTBwm/rJPUDCN/XJxdh5tL1xczWmVmzmbUAD9Lxl9SuA+rM7I1w+0mCX9jZeE7S9iULz0lU6lpU2XheWu3Wlyw8L2cBy82sPlw77Cngi0CfcFoN2nH9L082B0BSkaTi1p+BsQTLIkwFLg2rXQr8VzIRZm5vfWn9RRD6OkH/Oiwz+wBYLWlIWDSaYHmJrDsne+tLtp2TFKlrUWXdeYnYrS9ZeF5WASMl9ZAkPv238hJwflin3c6J3412ABQsVf10uJkH/MHMfirpUII1dI4kOKEXmNlHCYWZkX305XcE0wIGrAC+3zrH3lFJGgb8CiggWN31MoI/rLLqnMBe+3I3WXZOYNdaVKuBQWa2MSzLun8rsNe+ZOO/lf8NfBtoAv4B/DeCazSPAn3DskvMbOcBf5YnG+ecc3HzaTTnnHOx82TjnHMudp5snHPOxc6TjXPOudh5snHOORc7TzbOOedi58nGuSwh6SFJ57dd07mOx5ONc8652Hmyce4ASCoPFzV7MFyEapqk7pJmSKoO65SEawUh6V8k/UnSnyUtlzRB0sTwqc6zJPXN8HNPlPRy+ITu5yLPF/tXSbMlzZP0x/BRJL0VLIyXE9bpIWm1pHxJgyU9Gx7nVUmfC+tcEC6oNU/SK7H8x3Ndiicb5w5cJXCfmR0DfAJ8s436xwIXETyo8afAtvCpzn8HvtvWh0nKB+4BzjezE4HfhMeBYNGrL5jZ54FFwOXh41TmAaeHdf4ZeC58+OIDwNXhcW4A/iOs8xPg7PA449qKybm25LVdxTnXhuVm9lb48xygvI36L5nZZmCzpI3An8PyBcDxGXzeEIKENT14fiK5BI+0BzhW0v8B+gA9gefC8scInoH1EsEaLP8hqSfBU36fCI8DUBi+zwQeCldvfCqDmJzbJ082zh246EMKm4HuBA82bJ056LaP+i2R7RYy+zcpYKGZnZxm30PA18xsnqR/Ac4Iy6cC/zecpjsReJFg/ZJPzGxY6kHM7EpJJwFfAd6SNMzMPswgNufS8mk05+KxguCXOnz6uPb2shgolXQyBNNqko4J9xUDa8OptotbG5jZFuBN4C7gmXDdlU3AckkXhMeRpM+HPw82szfM7CfABqCsnfvguhhPNs7FYzLwA0mvAyXteWAzayBIYP9P0jzgLYLpMID/BbwBTAfeTWn6GHBJ+N7qYuDy8DgLCVZuBPi5pAWS3gZeIbjm49xn5ksMOOeci52PbJxzzsXObxBwroORdB9wSkrxXWY2JYl4nGsPPo3mnHMudj6N5pxzLnaebJxzzsXOk41zzrnYebJxzjkXu/8PFSqdm5eIIEUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_means, yerr=test_stds,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_means, yerr=train_stds,label = ' Train')\n",
    "plt.xlabel( 'num_leaves' )\n",
    "plt.ylabel( 'Log Loss' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.48760818, -0.48760542, -0.48720983, -0.48721939])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 取系统推荐值：70, 不必再细调"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. min_child_samples\n",
    "叶子节点的最小样本数目\n",
    "\n",
    "叶子节点数目：70，共9类，平均每类8个叶子节点\n",
    "每棵树的样本数目数目最少的类（稀有事件）的样本数目：200 * 2/3 * 0.7 = 100\n",
    "所以每个叶子节点约100/8 = 12个样本点\n",
    "\n",
    "搜索范围：10-50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  7.7min remaining:  3.9min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed: 11.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.7,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'min_child_samples': range(10, 50, 10)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(10,50,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4864051545458686\n",
      "{'min_child_samples': 40}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAdYklEQVR4nO3deZRcdZ338fe3t3R30tlId2dPJxiSdBIWKUJcQAQSE4REPPgcQB19HMV5jhyVwVFwRVBURgecZzjzTAZFnQXkUcCIj5CwREEF0mExSQdIJ5AFku7OvnZ6+z5/1K1Opemleqm+VXU/r3NyknvrVvfv58X+dP1ufW6ZuyMiItGTF/YAREQkHAoAEZGIUgCIiESUAkBEJKIUACIiEVUQ9gD6Yty4cV5VVRX2MEREssq6dev2uHt55/1ZFQBVVVXU1NSEPQwRkaxiZtu62q8lIBGRiEopAMxsiZm9amZ1ZnZTD8ddZWZuZrFgu8jM7jWz9Wb2spldlHTsd81sh5kdGfAsRESkz3oNADPLB+4GlgLVwDVmVt3FcWXA54HnknZ/BsDd5wOLgB+ZWeJ7/hZYMKDRi4hIv6XyCmABUOfuW929GbgfWN7FcbcBdwBNSfuqgScA3L0BOADEgu1n3X3XAMYuIiIDkEoATAJ2JG3vDPZ1MLNzgCnu/kin574MLDezAjObDpwLTOnLAM3sOjOrMbOaxsbGvjxVRER6kMq7gKyLfR13kAuWdO4EPtnFcT8F5gA1wDbgz0BrXwbo7iuAFQCxWEx3rhMRGSSpBMBOTv2tfTLwVtJ2GTAPWGNmAOOBlWa2zN1rgBsSB5rZn4HNAx20iIgMXCpLQGuBmWY23cyKgKuBlYkH3f2gu49z9yp3rwKeBZa5e42ZlZrZcAAzWwS0unvt4E+je+3tzi/XbuexjbuH8tuKiGS8Xl8BuHurmV0PPAbkAz91941mditQ4+4re3h6BfCYmbUDbwIfTzxgZncA1wKlZrYTuMfdb+n/VLoZP/Cfz25n96EmFs44jVElhYP9LUREspJl0wfCxGIx708TeP3Ogyy/+xmuPX8q3/nQ/DSMTEQkc5nZOnePdd4fiSbw/Mmj+OS7p/Nfz21n3bb9YQ9HRCQjRCIAAP5+8RmMH1nM1x5aT0tbe9jDEREJXWQCYMSwAr69bC6v7D7MT555PezhiIiELjIBALB47ngWV1dy1+OvsWPfsbCHIyISqkgFAMAty+aSb8bXH95ANl0AFxEZbJELgImjS7hx8Sz+8Fojv1uvWxGJSHRFLgAAPvHuKuZPGsW3f1vLweMtYQ9HRCQUkQyA/Dzj9ivns/fICf7xsVfCHo6ISCgiGQCgboCISGQDANQNEJFoi3QAjBhWwK3L56kbICKRFOkAAFhUXckH5qobICLRE/kAAHUDRCSaFADAhFElfOkD8W7AI39VN0BEokEBEPibd6kbICLRogAI5OcZ3/vwfPYdPcEdj6obICK5TwGQZN6kUfzP96gbICLRoADo5O8XncHEUcV89UF1A0QktykAOhk+rIBvL5/Hq/WHuedpdQNEJHcpALqQ6Ab8+InX2L5X3QARyU0KgG50dAN+o26AiOQmBUA3Et2AP6obICI5SgHQA3UDRCSXKQB6oG6AiOQyBUAv1A0QkVylAEiBugEikosUAClQN0BEcpECIEXqBohIrlEA9MEty+ZSkJenboCI5AQFQB9MGFXClxafoW6AiOQEBUAfffxdVZw5Wd0AEcl+CoA+ys8zbr9S3QARyX4KgH6YN2kUn+roBuwLezgiIv2iAOinGzq6ARvUDRCRrKQA6Kfhwwq4NegG/PvTW8MejohIn6UUAGa2xMxeNbM6M7uph+OuMjM3s1iwXWRm95rZejN72cwuSjr23GB/nZn9s5nZgGczxC6trmTJ3PH8+PHN6gaISNbpNQDMLB+4G1gKVAPXmFl1F8eVAZ8Hnkva/RkAd58PLAJ+ZGaJ7/mvwHXAzODPkv5PIzy3LJtLYb66ASKSfVJ5BbAAqHP3re7eDNwPLO/iuNuAO4CmpH3VwBMA7t4AHABiZjYBGOnuf/H4T81fAB/q/zTCM35UcUc34LfqBohIFkklACYBO5K2dwb7OpjZOcAUd3+k03NfBpabWYGZTQfOBaYEz9/Z09dM+trXmVmNmdU0NjamMNyhl+gG3PrbWg4eUzdARLJDKgHQ1dp8x1pHsKRzJ3BjF8f9lPgP9xrgLuDPQGtvX/OUne4r3D3m7rHy8vIUhjv0krsBP3hM3QARyQ6pBMBO4r+1J0wG3kraLgPmAWvM7A1gIbDSzGLu3uruN7j72e6+HBgNbA6+5uQevmbWSXQD/lvdABHJEqkEwFpgpplNN7Mi4GpgZeJBdz/o7uPcvcrdq4BngWXuXmNmpWY2HMDMFgGt7l7r7ruAw2a2MHj3z98AvxnkuQ05dQNEJJv0GgDu3gpcDzwGbAIecPeNZnarmS3r5ekVwAtmtgn4CvDxpMf+F3APUAdsAX7fj/FnFHUDRCSbWDa9dTEWi3lNTU3Yw+jV3/3HOp56tYHVN7yPqaeVhj0cEYk4M1vn7rHO+9UETgN1A0QkGygA0kDdABHJBgqANPn4u6o4S90AEclgCoA0yc8zbv/wfPYfa1Y3QEQykgIgjeZOHMWn3lOlboCIZCQFQJp98dIzmDS6RN0AEck4CoA0i3cD5qobICIZRwEwBC6ZU8nSefrcABHJLAqAIfKtK+LdgK89vF7dABHJCAqAITJ+VDH/8IFZPL15Dytfzur73olIjlAADKGPLZzGWZNHcdsj6gaISPgUAEPoZDeghe8/qm6AiIRLATDEEt2A+57fTs0b6gaISHgUACHo6AY8tJ7mVnUDRCQcCoAQJLoBr9UfUTdAREKjAAhJohvwz09sZtveo2EPR0QiSAEQokQ34OsP63MDRGToKQBCpG6AiIRJARAydQNEJCwKgJCpGyAiYVEAZAB1A0QkDAqADKFugIgMNQVAhlA3QESGmgIgg1wyp5LL5qsbICJDQwGQYdQNEJGhogDIMJUji/nyEnUDRCT9FAAZ6KPnT+OsKaPVDRCRtFIAZKD8POP2K+epGyAiaaUAyFBzJ47ib987nfue385adQNEJA0UABnsi5fOjHcDHlQ3QEQGnwIgg5UWxbsBmxvUDRCRwacAyHDJ3YA39qgbICKDRwGQBRLdgG/8Rt0AERk8CoAsoG6AiKSDAiBLJHcDDhxrDns4IpIDUgoAM1tiZq+aWZ2Z3dTDcVeZmZtZLNguNLOfm9l6M9tkZjcnHfsFM9tgZhvN7IsDn0puS+4G/EDdABEZBL0GgJnlA3cDS4Fq4Bozq+7iuDLg88BzSbs/Agxz9/nAucBnzazKzOYBnwEWAGcBl5vZzIFOJted7AbsUDdARAYslVcAC4A6d9/q7s3A/cDyLo67DbgDaEra58BwMysASoBm4BAwB3jW3Y+5eyvwB+DK/k8jOtQNEJHBkkoATAJ2JG3vDPZ1MLNzgCnu/kin5/4KOArsArYDP3T3fcAG4EIzO83MSoHLgCldfXMzu87MasysprGxMZU55TR1A0RksKQSANbFvo73IppZHnAncGMXxy0A2oCJwHTgRjOb4e6bgB8Aq4FHgZeB1q6+ubuvcPeYu8fKy8tTGG7uUzdARAZDKgGwk1N/O58MJL8XsQyYB6wxszeAhcDK4ELwtcCj7t7i7g3An4AYgLv/xN3f6e4XAvuAzQOdTJR864q5FKkbICIDkEoArAVmmtl0MysCrgZWJh5094PuPs7dq9y9CngWWObuNcSXfS62uOHEw+EVADOrCP6eCnwYuG8Q55Xz1A0QkYHqNQCCi7TXA48Bm4AH3H2jmd1qZst6efrdwAjia/5rgXvd/a/BY782s1rgt8Dn3H1/fycRVdeeP42z1Q0QkX6ybFo+iMViXlNTE/YwMkrtW4e44l+e4X/EJvO9D58Z9nBEJAOZ2Tp3j3XeryZwlqueOJJPqxsgIv2gAMgBX1A3QET6QQGQA0qLCvjOh+apGyAifaIAyBHvn13BB+dPUDdARFKmAMgh37yimqL8PL7+sLoBItI7BUAOSXQDnqnbw29eUjdARHqmAMgx6gaISKoUADkm/rkB8zlwvIXv/16fGyAi3VMA5KBEN+D+tTt4/nV1A0SkawqAHNXRDXhI3QAR6ZoCIEclugF1DUdY8cctYQ9HRDKQAiCHdXQDnqxTN0BE3kYBkOO+eUU1w9QNEJEuKABynLoBItIdBUAEqBsgIl1RAERAfp7xvQ+rGyAip1IARMScCSP59AXqBojISQqACPnCJTOZPEbdABGJUwBESGlRAbepGyAiAQVAxLx/VgUfPFPdABFRAETSty5XN0BEFACRVDGymC8vna1ugEjEKQAi6qMLpqobIBJxCoCIylM3QCTyFAARpm6ASLQpACIu0Q24+cG/cqK1LezhiMgQUgBEXKIbsKXxKCv+sDXs4YjIEFIASEc34H8/Vcfr6gaIRIYCQIDkbsB6dQNEIkIBIMDJbsCf6vby8Etvhj0cERkCCgDpcLIbsIn9R9UNEMl1CgDpkOgGHFQ3QCQSFAByikQ34Jc1O3hu696whyMiaaQAkLdJ/twAdQNEcpcCQN5G3QCRaEgpAMxsiZm9amZ1ZnZTD8ddZWZuZrFgu9DMfm5m681sk5ndnHTsDWa20cw2mNl9ZlY88OnIYHn/rAouVzdAJKf1GgBmlg/cDSwFqoFrzKy6i+PKgM8DzyXt/ggwzN3nA+cCnzWzKjObFBwbc/d5QD5w9UAnI4Prm5dXM6xA3QCRXJXKK4AFQJ27b3X3ZuB+YHkXx90G3AE0Je1zYLiZFQAlQDNwKHisACgJHisFdGP6DFMxspivLFE3QCRXpRIAk4AdSds7g30dzOwcYIq7P9Lpub8CjgK7gO3AD919n7u/Cfww2LcLOOjuq7r65mZ2nZnVmFlNY2NjKnOSQXTtgqmcM1XdAJFclEoAWBf7OtYDzCwPuBO4sYvjFgBtwERgOnCjmc0wszHEX0VMDx4bbmYf6+qbu/sKd4+5e6y8vDyF4cpgSnQDDqkbIJJzUgmAncCUpO3JnLpcUwbMA9aY2RvAQmBlcCH4WuBRd29x9wbgT0AMuBR43d0b3b0FeBB490AnI+kxe/xIPn3BDHUDRHJMKgGwFphpZtPNrIj4xdqViQfd/aC7j3P3KnevAp4Flrl7DfElnostbjjxcHgl2L/QzErNzIBLgE2DOjMZVOoGiOSeXgPA3VuB64HHiP+QfsDdN5rZrWa2rJen3w2MADYQD5J73f2v7v4c8esDLwDrg3Gs6P80JN1KivLVDRDJMZZNb++LxWJeU1MT9jAi7fr/foFVtfU89sULmT5ueNjDEZEUmNk6d4913q8msPRJohvwtYfUDRDJdgoA6ZNEN+DPW/by0IvqBohkMwWA9FmiG/Cd36kbIJLNFADSZ8ndgO/9Xm/eEslWCgDpl0Q34IGanTyrboBIVlIASL8lugFfUzdAJCspAKTfkrsB/6ZugEjWUQDIgCQ+N+Bfnqpja+ORsIcjIn2gAJAB++YVic8N2KBugEgWUQDIgFWUFXPTUnUDRLKNAkAGxTXnTeWd6gaIZBUFgAyKvDzjdnUDRLKKAkAGzezxI/nMheoGiGQLBYAMqs9fPJMpY9UNEMkGCgAZVCVF+dy2XN0AkWygAJBBd9GsCq44a6K6ASIZTgEgafGNy+eoGyCS4RQAkhbqBohkPgWApI26ASKZTQEgaaNugEhmUwBIWqkbIJK5FACSdoluwFfVDRDJKAoASbtEN2Br41H+zxp1A0QyhQJAhkSiG3C3ugEiGUMBIEPmG5fPYVhhHl97SN0AkUygAJAhk+gG/GXrXh58Qd0AkbApAGRIXXPeVM6dNobv/K6WfeoGiIRKASBDKi/PuP3K+RxuauWWlRvZ+NZBtjYeYdfB4xw41kxTS5uWh0SGSEHYA5DomTW+jM++bwZ3P7WFlS+/9bbHzaCkMD/+p+jk38WJfUnbpV09XpRHSWHi8YKOfcXJzy3IJy/PQpi9SOZQAEgoblw0iwtmlnPweAtNLW0ca27jeHMbx1vaaGo5+e/k7WPNbRw41syuYP/x5vbgua209+NFw7CCvI4AKS56e7icEkJJj596bB4lhQWdHs/r+HdBvl5kS+ZSAEgo8vKMhTNOG5Sv5e60tPkpodFlmHR6vKnTsYm/j5xopfHwiVMeb2ppp7mtvc9jK8y3U17JFBe+PUxKU3w8+RVMctgU5edhplcz0ncKAMl6ZkZRgVFUkMcoCtP2fVra2jtCoam5/W1hciz5VUsXYXOsUygdPN5yyrHHmts40dr3kMnPs6TwOPnq45Qw6e4VTufHi/I5o6KMUaXp+99RMocCQCRFhfl5FObnUVacvh+O7e1OU2vnVzDJYdPasfzV1XLZ8eZTt/cdbT7layVCqKfr7AV5xoLpY1lUXcmi6komjylN23wlXJZN77iIxWJeU1MT9jBEspq7c6K1/W1LYE0tbRxuauX51/exqraeuoZ4Y7t6wsiOMJg7caSWm7KQma1z99jb9isARKQrr+85yura3ayuradm237cYdLoEi6dU8HiueNZMH0shbrInRUGFABmtgT4MZAP3OPu3+/muKuA/wuc5+41ZlYI3AO8k/hy0y/c/XtmNgv4ZdJTZwDfdPe7ehqHAkAkHHuOnODJTQ2sqq3nmbpGmlraGVlcwPtnV7CoupL3nVGe1qUxGZh+B4CZ5QOvAYuAncBa4Bp3r+10XBnwO6AIuD4IgGuBZe5+tZmVArXARe7+Rqev/yZwvrtv62ksCgCR8B1vbuPpzY2srq3niVca2He0maL8PBaefhqLg6WiypHFYQ9TknQXAKlcBF4A1Ln71uAL3Q8sJ/7DPNltwB3Al5L2OTDczAqAEqAZONTpeZcAW3r74S8imaGkKJ/Fc8ezeO542tqdddv2dywVff3hDXz94Q2cNXlUcN1gPGdUjtB1gwyVSgBMAnYkbe8Ezk8+wMzOAaa4+yNmlhwAvyIeFruAUuAGd9/X6etfDdzX3Tc3s+uA6wCmTp2awnBFZKjkB+8YWjB9LF+9bA51DUdYVVvPqtp6frjqNX646jWmnVbKojnxVwaxqrHkq4GdMVIJgK7OVse6kZnlAXcCn+ziuAVAGzARGAM8bWaPJ72aKAKWATd3983dfQWwAuJLQCmMV0RCYGbMrCxjZmUZn3v/O6g/1MTjm+pZXVvPL/6yjXueeZ0xpYVcPLuSxXMruWDmOEqL9E70MKXyv/5OYErS9mQg+QYuZcA8YE3wMm88sNLMlgHXAo+6ewvQYGZ/AmJA4mOhlgIvuHv9gGYhIhmncmQxHz1/Gh89fxpHTrTyh1cbg6Wi3fz6hZ0MK8jjgpnjWFRdySVzKhk3YljYQ46cVAJgLTDTzKYTv1h7NfEf7AC4+0FgXGLbzNYAXwouAl8CXGxm/0l8CWghkPxOn2voYflHRHLDiGEFfPDMCXzwzAm0tLWzNugarK6t5/FNDZit59ypYzr6BjPKR4Q95EhI9W2glxH/wZ0P/NTdv2tmtwI17r6y07FrOBkAI4B7gWriS0n3uvs/BseVEr+2MCMIkV7pXUAiucXdqd11iNVBGGx8K/4ekdPLh7OoejyL51Zy9uTRunPrAKkIJiIZb+f+YzxeW8/qTfU8t3Ufre1OedkwLp0T7xu8+/RxFBfmhz3MrKMAEJGscvBYC2tea2DVxnrWvNrA0eY2SovyuXBmOYvnVnLx7ApGlxaFPcysoAAQkax1orWNv2zZG1wzqKf+0Any84zzqsbEl4qqK5kyVjet644CQERyQnu7s/7Ng6wKymev1cdvWjd7fFnQRB7PvEm6aV0yBYCI5KRte4+yOiif1byxj3aHCaOKuXROvG9w/vTTKCqI9k3rFAAikvP2HW3miaB89sfN8ZvWlQ0r4KLgpnUXzSpnZARvWqcAEJFIaWpp45nNe1hVu5snNjWw92gzhfnxjyJdXF3JpdWVTBhVEvYwh4QCQEQiq63deXH7/o6+wdY9RwGYPyl+07rFcyuZVVmWs9cNFAAiIoH4TeviF5Ff3H4AgCljS1g0ZzyLqis5r2oMBTn0YTcKABGRLjQcbuKJTQ2srq3nmbo9NLe2M7q0kItnV7C4upILZpYzfFh237ROASAi0oujJ1r542snP+zm4PEWigryeO87Ejetq6CiLPs+7GYgHwgjIhIJw4cVsHT+BJbOn0BrWzvPv7Gv47rBk680YAbnTBnNour4UtE7KrL7pnV6BSAi0gt355XdhzvCYP2b8ftXzhg3vOMi8tlTxmTsh91oCUhEZJC8deB4x4fd/GXLXlrbnXEjirhkdvx21u+dmVk3rVMAiIikwaGmFta82siqjbv5w6uNHD7RSklhPheeMY5F1eO5eHYFY4eHe9M6XQMQEUmDkcWFLDtrIsvOmkhzazvPbt3bsVT02MZ68gxiVWOD+xRVMu204WEPuYNeAYiIpIF7/KZ1iTB4ZfdhAGZVlnV88tn8SaOG5MNutAQkIhKi7XuPsXpTPas27mZtcNO6ypHDgjAYz7tmpO+mdQoAEZEMsf9oM0++0tBx07pjzW2MGFbA+2aVs7i6kotmVTCqZPBuWqcAEBHJQE0tbfx5yx5Wbazn8U0N7DlygoK8+E3rEktFE0cP7KZ1CgARkQzX3u68uONAcN1gN1sa4zetmztxJD//1ALGjRjWr6+rdwGJiGS4vDzj3GljOHfaGG5aOpstjUdYXVvPC9v2c1oa3kqqABARyVCnl4/g9Pel73YTuXO/UxER6RMFgIhIRCkAREQiSgEgIhJRCgARkYhSAIiIRJQCQEQkohQAIiIRlVW3gjCzRmBbP58+DtgziMMJU67MJVfmAZpLpsqVuQx0HtPcvbzzzqwKgIEws5qu7oWRjXJlLrkyD9BcMlWuzCVd89ASkIhIRCkAREQiKkoBsCLsAQyiXJlLrswDNJdMlStzScs8InMNQEREThWlVwAiIpJEASAiElE5GQBm9lMzazCzDUn7xprZajPbHPw9JswxpqKbedxiZm+a2UvBn8vCHGOqzGyKmT1lZpvMbKOZfSHYn1XnpYd5ZN15MbNiM3vezF4O5vLtYP90M3suOCe/NLPB/yiqQdbDXH5mZq8nnZezwx5rKsws38xeNLNHgu20nJOcDADgZ8CSTvtuAp5w95nAE8F2pvsZb58HwJ3ufnbw5/8N8Zj6qxW40d3nAAuBz5lZNdl3XrqbB2TfeTkBXOzuZwFnA0vMbCHwA+JzmQnsB/42xDGmqru5APxD0nl5Kbwh9skXgE1J22k5JzkZAO7+R2Bfp93LgZ8H//458KEhHVQ/dDOPrOTuu9z9heDfh4n/xz2JLDsvPcwj63jckWCzMPjjwMXAr4L9GX9OoMe5ZB0zmwx8ELgn2DbSdE5yMgC6UenuuyD+f2KgIuTxDMT1ZvbXYIkoo5dMumJmVcA5wHNk8XnpNA/IwvMSLDW8BDQAq4EtwAF3bw0O2UmWBFznubh74rx8Nzgvd5rZsBCHmKq7gC8D7cH2aaTpnEQpAHLFvwKnE3+Zuwv4UbjD6RszGwH8Gviiux8Kezz91cU8svK8uHubu58NTAYWAHO6OmxoR9U/nediZvOAm4HZwHnAWOArIQ6xV2Z2OdDg7uuSd3dx6KCckygFQL2ZTQAI/m4IeTz94u71wX/o7cC/E/8/bVYws0LiPzT/y90fDHZn3Xnpah7ZfF4A3P0AsIb4dY3RZlYQPDQZeCuscfVH0lyWBEt27u4ngHvJ/PPyHmCZmb0B3E986ecu0nROohQAK4FPBP/+BPCbEMfSb4kfloErgQ3dHZtJgnXMnwCb3P2fkh7KqvPS3Tyy8byYWbmZjQ7+XQJcSvyaxlPAVcFhGX9OoNu5vJL0y4URXzfP6PPi7je7+2R3rwKuBp5094+SpnOSk01gM7sPuIj4LVTrgW8BDwMPAFOB7cBH3D2jL7B2M4+LiC8zOPAG8NnEGnomM7P3Ak8D6zm5tvlV4uvnWXNeepjHNWTZeTGzM4lfUMwn/svgA+5+q5nNIP7b51jgReBjwW/QGauHuTwJlBNfRnkJ+Luki8UZzcwuAr7k7pen65zkZACIiEjvorQEJCIiSRQAIiIRpQAQEYkoBYCISEQpAEREIkoBICISUQoAEZGI+v90CiAvvt4Q7QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.49089884, -0.48720983, -0.48711861, -0.48640515])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  6.9min remaining:  3.5min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed: 10.2min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.7,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'min_child_samples': range(40, 80, 10)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(40,80,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.485439275836877\n",
      "{'min_child_samples': 50}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhV5bX48e/KPJCEKQyZCGGUmRACiCIyidWCVVRAWzv86giiYq3e9nbQtreXqohKUVul3KogUq1UrTJX1DIkzGMIISQhDGEKY8jA+v1xdmjAQA6QZOck6/M8PGS/ezjrfXZy1ln7Pe/eoqoYY4wx5fzcDsAYY0zdYonBGGPMeSwxGGOMOY8lBmOMMeexxGCMMeY8AW4HUB2aN2+uiYmJbodhjDE+JT09/aCqRl/YXi8SQ2JiImlpaW6HYYwxPkVEdlfWbpeSjDHGnMcSgzHGmPNYYjDGGHMeSwzGGGPOY4nBGGPMeSwxGGOMOY8lBmOMMeexxGCMqdKGvKPMS8/jxJlSt0MxtaBeTHAzxtScM6VlPPT2GvYcPc0vP9rEqF6xjE9NoHtclNuhmRpiicEYc0lzV+ey5+hp/vvWLmzbe4wP1+Yxe1UO3WIjGZeawKieMUSEBLodpqlGUh+e4JaSkqJ2Swxjql9RSRk3/GEpCU3DmPvAAESEY0UlfLR2D++szGHbvuOEBfkzqmcM41IT6BEXhYi4Hbbxkoikq2rKhe1WMRhjLurtFbvZf+wM08b2PveGHxkSyHcHJHJv/zaszytk9socPlqXz5zVuXRpHcm4fgmM7hVDpFURPssqBmNMpU4VlzJoylI6tYrgnf/X/5LbHi8q4aN1+by7Mocte48RGuhUEf0S6GlVRJ1lFYMx5rLM+no3B08U8/rwTlVuGxESyL3923BPvwQ25BUye1UO89fn815aLte0jmR8ajyje8daFeEjrGIwxnzD8aISrp+ylN7xjZn5g9QrPsb89Z4qYnO+p4q4tUdrxvVLoHd8Y6si6gCrGIwxXnvry2yOnirhCS+qhYuJCAnknn5tuKdfGzbmFfLuqhzmr9vD++l5dG4Vwfh+CYzuFUtUqFURdY1VDMaY8xw9Vcz1/7uUAe2a8cb3vvFh8qqcOFPK/HX5zF6Vw8Y9hYQE+nFrD883mpITrIqobVYxGGO88qflWZwoLuWJER2r/diNggMY3y+B8f0S2JhXyOzVOXy0dg/z0vPo1DKCcanxfKd3HFFhVkW4ySoGY8w5h06c4fopSxnSuQWvjk+uldc8eaaUf6z3VBHr8woJDvBUEeP7xZOc0MSqiBp0sYrBq3slichIEdkuIpki8vQlthsjIioiKc5yoIjMEpGNIrJVRJ6psG1jEZknItucdQOc9l4iskJE1olImohc2ciXMeayvfavnRSVlPHYsOqvFi4mPDiAsakJfDThOj6eeB1j+sTx+eZ93DHj39z00hfM/GoXhadKai0e40XFICL+QAYwHMgDVgPjVHXLBdtFAJ8AQcAEVU0TkfHAKFUdKyJhwBZgsKpmi8gsYLmq/llEgoAwVT0qIguAqar6TxH5FvCUqg6+VIxWMRhz9Q4cK+L6KUu5pUdrXryrl6uxnDxTyscb8nl3VS7rc48SHODHLd0932hKaWNVRHW5mjGGVCBTVbOcA80BRuN5k6/oOWAK8GSFNgXCRSQACAWKgWMiEgkMAr4PoKrFzrryfSKdn6OAfC9iNMZcpT8u20npWWXS0A5uh0J4cAB3903g7r4JbM4vZM6qXP6+dg8frN1D+xaNGJeawB3JsTQOC3I71HrJm0tJsUBuheU8p+0cEekNxKvqxxfsOw84CewFcoDnVfUwkAQUADNFZK2I/FlEwp19HgP+ICK5wPPAM1RCRO53LjWlFRQUeNENY8zF7Dl6mndX5nBXShxtmoVXvUMt6hoTxXO3dWPlz4YyZUwPIkICeO7jLaT+bjGPzVnLql2HqQ9jpXWJN4mhsprt3FkQET9gKjC5ku1SgTIgBmgLTBaRJDyVSjIwQ1V740ke5WMXDwGPq2o88DjwZmVBqeobqpqiqinR0dFedMMYczGvLskEYMIQ96uFiwkLCuCulHg+fHgg/5x0PeP6xrN42wHuev3fDHvxX/x5eRZHThZXfSBTJW8SQx4QX2E5jvMv70QA3YBlIpIN9AfmOwPQ44HPVLVEVQ8AXwEpzjHzVHWlc4x5eBIFwH3AB87P7+NJLsaYGpJz6BTvp+UyLjWe2MahbofjlWtaR/Lr0d1Y9V/D+MOYHkSFBvKbT7bS73eLmTRnLSuyDlkVcRW8GWNYDXQQkbbAHmAsnjd8AFS1EGheviwiy4AnncHnocAQEXkbCMOTNF5S1X0ikisinVR1OzCU/4xZ5AM3AMuAIcCOq+uiMeZSpi3egb+f8MiN7d0O5bKFBvlzZ0o8d6bEs23fMeasyuVva/L4aF0+SdHhjOubwB194mgabmMRl6PKxKCqpSIyAfgc8AfeUtXNIvIskKaq8y+x+3RgJrAJzyWpmaq6wVk3EXjH+UZSFvADp/3HwDRnwLoIuP8K+mWM8cLOghN8uDaPHw5sS4vIELfDuSqdW0Xyq1Fd+enIzny6cS/vrsrht59u5Q+fb2dkt1aMS02gf1JT+0aTF2yCmzEN2MTZa1m8dT9fPHUjzRsFux1Otdu+7zizV+XwwZo8jhWV0rZ5OONS47kjOY5m9bC/l+tiX1e1xGBMA7Vt3zFunrach25ox1MjO7sdTo0qKinj0417mb0qh9XZRwj0F27q2orxqQkMaNeswVYRdq8kY8x5pi7MoFFQAPcPSnI7lBoXEujP7clx3J4cx479x5ntjEV8vGEvic3CPPMi+sTVy6rpSljFYEwDtGlPIbe+8iWPDetQq7e/qEuKSsr4bNM+3l2Zw6rswwT6CyPKq4ikZvj51f8qwioGY8w5Ly7MICo0kB9e19btUFwTEujPbb1jua13LJkH/lNFfLJhL22ahTG2bwJj+sQRHdHwqgirGIxpYNJ3H+GOGV/z1MhOPDzY976iWpOKSsr4fLOnili56zABfsKIri0Zl5rAwHbN610VYRWDMQbwjC00Cw/ivgGJbodS54QE+jO6Vyyje8WSeeAE763OYV56Hp9u3EdC0zDGpsYzpk8cLSJ8+6u9VbGKwZgGZEXWIca+sYKf33IN/+/6+j/oXB3OlJbx+eb9vLtyNyuyPFXE8C6eKuK69r5dRVjFYEwDp6q8uCCDlpHB3Nu/jdvh+IzgAH9G9YxhVM8YsgpOMGd1LvPS8/jnpn3ENw1lbN8E7uwT5/MTBCuyisGYBmL5jgK+++Yqnh3dle/ZZaSrcqa0jAWb9zN7VQ5f7zyEv58w7JoWjEtNYFCHaJ+pIqxiMKYBU1WeX5BBbONQ7u4bX/UO5pKCA/z5ds8Yvt0zhl0HTzJnVQ7vp+fx+eb9xDYOZVyq5/5NLX20irCKwZgGYPHW/fxoVhq/v707Y1MT3A6nXjpTWsbCLZ4q4qtMTxUxtHMLxvXzVBH+dbCKsIrBmAbq7FnlxYUZtGkWxh194twOp94KDvDn1h4x3NojhuyDJ52xiFwWbNl/rlK7KyWeVlF1v4qwisGYeu6fG/fy0DtrePGuntyebImhNhWXnmXR1v28uzKHLzMP4u8nDOncgvGpCQzq6H4VYRWDMQ1Q2Vll6qIM2kWHM7pXbNU7mGoVFODHt7q35lvdW7P7kKeKeD8tl4Vb9hMTFcLdfRO4q28craPq1gOSrGIwph77aN0eJs1Zx6vje3Nrjxi3wzF4qojFW/fz7qoclu84iJ/AkM6ebzQN7tSiVqsIqxiMaWBKy87y0qIddG4Vwbe6tXY7HOMICvDj5u6tubl7a3IOneK9tBzmpuWxaGsaraNCzo1FxLj4mFWrGIypp95Py+Un8zbw+nf7cFPXVm6HYy6hpOwsi7ceYPaqHL7YUYAAN3YqryKiCfD3q5HXtYrBmAakuPQs0xbvoHtsFCO6tHQ7HFOFQH8/RnZrxchurcg9fIr3VufyXloui/8vjVaRIdzVN567+8YTW0tVhFdpSERGish2EckUkacvsd0YEVERSXGWA0VklohsFJGtIvJMhW0bi8g8EdnmrBtQYd1E5/U2i8iUq+mgMQ3R++m55B05zRMjOjbYp5P5qvimYTx5Uye+fnoIr3+3D51bR/DKkh1c979L+MHMVSzYvI/SsrM1GkOVFYOI+APTgeFAHrBaROar6pYLtosAHgVWVmi+EwhW1e4iEgZsEZHZqpoNTAM+U9UxIhIEhDnHuREYDfRQ1TMi0uKqe2lMA1JUUsYrizPp06YJgztGux2OuUKB/n7c1LUVN3VtRd6RU8x1qoj7/5pOy8hg7k6J566+8cQ1Cav21/amYkgFMlU1S1WLgTl43rgv9BwwBSiq0KZAuIgEAKFAMXBMRCKBQcCbAKparKpHnX0eAn6vqmecdQcuv1vGNFyzV+Ww71gRk4dbtVBfxDUJ44kRnfjqp0P40/dS6NI6kleWZnL9lKV8vnlftb+eN4khFsitsJzntJ0jIr2BeFX9+IJ95wEngb1ADvC8qh4GkoACYKaIrBWRP4tIuLNPR+B6EVkpIv8Skb6VBSUi94tImoikFRQUeNENY+q/08VlTF+6k/5JTbm2fXO3wzHVLMDfj+FdWjLzB6l8+dMhTBzSgf5Jzar9dbxJDJV95Dj3VSYR8QOmApMr2S4VKANigLbAZBFJwnMJKxmYoaq98SSP8rGLAKAJ0B/4CTBXKvnYo6pvqGqKqqZER1u5bAzA//07m4MnzjB5RCe3QzE1LLZxKE8M70hUaGC1H9ubxJAHVLwdYxyQX2E5AugGLBORbDxv6POdAejxeMYRSpxLQl8BKc4x81S1fDxiHp5EUf56H6jHKuAsYB99jKnCiTOlvPavnQzqGE3fxKZuh2N8mDeJYTXQQUTaOoPEY4H55StVtVBVm6tqoqomAiuAUaqahufy0RDxCMeTNLap6j4gV0TKP9YMBcoHs/8ODAEQkY5AEHDwajtqTH0388tdHDlVwuThHd0Oxfi4Kr+VpKqlIjIB+BzwB95S1c0i8iyQpqrzL7H7dGAmsAnPJamZqrrBWTcReMdJNlnAD5z2t4C3RGQTnsHq+7Q+zMIzpgYVni7hT8uzGHZNS3rGN3Y7HOPjvJrgpqqfAp9e0PaLi2w7uMLPJ/B8ZbWy7dbhuax0YXsxcK83cRljPN5cnsWxolKesGrBVIOamWdtjKk1h08W8+aXu7ile2u6xES6HY6pBywxGOPjXv9iJ6dKynhsWAe3QzH1hCUGY3zYgeNFzPo6m9t6xdKhZYTb4Zh6whKDMT5sxrKdlJQpk4ZatWCqjyUGY3zU3sLTvLMyhzuSY0lsHl71DsZ4yRKDMT7q1SWZqCoTh1i1YKqXJQZjfFDu4VPMTcvl7r7xxDet/rtrmobNEoMxPujlxTsQESbcaNWCqX6WGIzxMbsOnuSDtXu4t18bWkWFuB2OqYcsMRjjY6YtyiDI34+HBrdzOxRTT1liMMaHZOw/zkfr87nv2kSiI4LdDsfUU5YYjPEhLy3KIDwogAcGJbkdiqnHLDEY4yM25xfy6cZ9/HBgIk3Cg9wOx9RjlhiM8RFTF2YQGRLAj663asHULEsMxviAdblHWbT1APcPSqqRRzkaU5ElBmN8wAsLttM0PIjvD2zrdiimAbDEYEwdtzr7MMt3HOTBG5JoFOzVs7WMuSqWGIypw1SV5z/fTnREMN/tn+h2OKaB8CoxiMhIEdkuIpki8vQlthsjIioiKc5yoIjMEpGNIrJVRJ6psG1jEZknItucdQMuONaTzrGaX2nnjPF1X+88xMpdh3lkcDtCg/zdDsc0EFUmBhHxB6YDNwNdgHEi0qWS7SKAR4GVFZrvBIJVtTvQB3hARBKdddOAz1S1M9AT2FrhWPHAcCDn8rvkvV0HTzIvPa8mX8KYK6aqvLBgO62jQhibmuB2OKYB8aZiSAUyVTVLVYuBOcDoSrZ7DpgCFFVoUyBcRAKAUKAYOCYikcAg4E0AVS1W1aMV9psKPOXsX2P+uDSTp+at5987D9XkyxhzRZZtL2BNzlEmDulASKBVC6b2eJMYYoHcCst5Tts5ItIbiFfVjy/Ydx5wEtiL59P/86p6GEgCCoCZIrJWRP4sIuHOsUYBe1R1/aWCEpH7RSRNRNIKCgq86MY3/XJUV9o2D2fi7DXsKyyqegdjaomq8uLCDOKbhnJnSpzb4ZgGxpvEIJW0nfskLyJ+eD7hT65ku1SgDIgB2gKTRSQJCACSgRmq2htP8nhaRMKAnwG/qCooVX1DVVNUNSU6OtqLbnxTo+AAXv9uH04Vl/HIu2soLj17Rccxprot2LKfjXsKeXRIBwL97TsipnZ58xuXB8RXWI4D8issRwDdgGUikg30B+Y7A9Dj8YwjlKjqAeArIMU5Zp6qlo9HzMOTKNrhSSDrnWPFAWtEpNWVda9q7VtEMGVMD9J3H+F3n26tegdjatjZs8qLCzJIah7Od3rHVr2DMdXMm8SwGuggIm1FJAgYC8wvX6mqharaXFUTVTURWAGMUtU0PJePhohHOJ6ksU1V9wG5ItLJOcxQYIuqblTVFhWOlQckO9vXmFt7xPDDgW35y9fZfLRuT02+lDFV+mTjXrbvP86kYR0IsGrBuKDK3zpVLQUmAJ/j+ebQXFXdLCLPOuMBlzIdaARswpNgZqrqBmfdROAdEdkA9AJ+d4V9qBbPfKszfROb8PTfNpKx/7iboZgGrLTsLFMXZdCpZQTf7hHjdjimgRLVGv3iT61ISUnRtLS0qz7O/mNF3PLyl0SGBPDRhIFEhNg9aUzt+lt6HpPfX89r9yYzsltrt8Mx9ZyIpKtqyoXtVqdW0DIyhFfH92b34VP85P0N1IekaXxHSdlZpi3eQdeYSG7qWmPDasZUyRLDBfonNeOnIzvx2eZ9/Gl5ltvhmAZkXnoeOYdPMXlER0Qq+zKgMbXDEkMlfnx9Ejd3a8X/fradFVk2+c3UvDOlZbyyeAe94htzY6cWbodjGjhLDJUQEaaM6UGbZmFMeHct+4/Z5DdTs+asyiW/sIgnR3SyasG4zhLDRUSEBPLavX04eaaUR95ZQ0mZTX4zNaOopIzpSzNJbduUge2buR2OMZYYLqVjywj+d0wP0nYf4X8+3eZ2OKaeenvFbg4cP8Pk4Ta2YOoGSwxVGNUzhu9fm8hbX+3i4w35Ve9gzGU4eaaUPy7byfUdmtMvyaoFUzdYYvDCf33rGvq0acJT8zawwya/mWr0l6+zOXyymCeGd3Q7FGPOscTghaAAP6aPTyYsyJ8H307nxJlSt0My9cCxohLe+CKLoZ1b0DuhidvhGHOOJQYvtYoK4ZVxyWQfOsVT89bb5Ddz1d5cvovC0yU8btWCqWMsMVyGAe2a8dRNnfh04z7e/HKX2+EYH3bkZDFvfbmLkV1b0S02yu1wjDmPJYbLdP+gJEZ2bcX//HMbK23ym7lCbyzP4kRxqVULpk6yxHCZRIQ/3NmDhKZhTJi9lgM2+c1cpoMnzvCXr7L5do8YOrWKcDscY77BEsMVKJ/8dqKolEfetclv5vK8tmwnZ0rLmDSsg9uhGFMpSwxXqFOrCH5/R3dWZx/h9/+0yW/GO/uPFfHXFbu5PTmOdtGN3A7HmEpZYrgKo3vFct+ANrz55S4+2bDX7XCMD5i+NJOys8qkoVYtmLrLEsNV+tktXUhOaMxT89aTecAmv5mLyztyitmrcrirbzzxTcPcDseYi7LEcJWCAvyYfk8yIYH+PPj2Gk7a5DdzEa8uyUQQJtzY3u1QjLkkrxKDiIwUke0ikikiT19iuzEioiKS4iwHisgsEdkoIltF5JkK2zYWkXkiss1ZN8Bp/4PTtkFEPhSRxlfbyZrWOiqUV8b1JqvgBE/9zZ78Zr4p++BJ3k/PY3y/BGIah7odjjGXVGViEBF/YDpwM9AFGCciXSrZLgJ4FFhZoflOIFhVuwN9gAdEJNFZNw34TFU7Az2BrU77QqCbqvYAMoBn8AHXtm/OT27qzCcb9vLWV9luh2PqmJcX7yDQX3h4cDu3QzGmSt5UDKlApqpmqWoxMAcYXcl2zwFTgIpf7FcgXEQCgFCgGDgmIpHAIOBNAFUtVtWjzs8LVLX8eswKIO7yu+WOB29IYkSXlvzPp1tZnX3Y7XBMHZF54Dh/X7eH7w1IpEVkiNvhGFMlbxJDLJBbYTnPaTtHRHoD8ar68QX7zgNOAnuBHOB5VT0MJAEFwEwRWSsifxaR8Epe+4fAPysLSkTuF5E0EUkrKCjwohs1T0R4/q6exDcN45F31nDguE1+MzB10Q5CA/15YFCS26EY4xVvEkNlTw45dxFdRPyAqcDkSrZLBcqAGKAtMFlEkoAAIBmYoaq98SSP88YuRORnQCnwTmVBqeobqpqiqinR0dFedKN2RIYEMuPeZI4VlTDh3bU2+a2B27r3GJ9s2MsPBralWaNgt8MxxiveJIY8IL7CchxQ8Yk1EUA3YJmIZAP9gfnOAPR4POMIJap6APgKSHGOmaeq5eMR8/AkCgBE5D7gVuAe9cGR3M6tIvn97T1YteswUz6zyW8N2dSFGUSEBPDj661aML7Dm8SwGuggIm1FJAgYC8wvX6mqharaXFUTVTURz7jAKFVNw3P5aIh4hONJGttUdR+QKyKdnMMMBbaA5xtQwE+dY5yqnm7Wvtt6x/K9AW340/JdfLrRJr81RBvyjrJgy35+fH0SUWGBbodjjNeqTAzOQPAE4HM83xyaq6qbReRZERlVxe7TgUbAJjwJZqaqbnDWTQTeEZENQC/gd077q3iqkIUisk5EXrvcTtUVP7+lC73iG/OT99eTeeCE2+GYWvbiwgwahwXyg4GJbodizGURH7xS8w0pKSmalpbmdhiVyj96mltf+ZKm4UF89MhAwoMD3A7J1IL03Ye5Y8a/efrmzjx4g31F1dRNIpKuqikXttvM5xoW0/g/k9+e/mCjTX5rIF5YkEHzRkF8b0Abt0Mx5rJZYqgFA9s3Z/KITvxjfT5/+Trb7XBMDft650G+3nmIhwa3JyzIKkTjeywx1JKHbmjHsGta8ttPtpJmk9/qLVXlxQUZtIoM4Z5+CW6HY8wVscRQS/z8hBfu6klsk1AeeXcNBcfPuB2SqQFf7DhI2u4jPDKkPSGB/m6HY8wVscRQi6JCPU9+KzxdwsTZayi1yW/1iqrywoLtxDYO5e6U+Kp3MKaOssRQy65pHcnvvtOdFVmH+cPn290Ox1SjRVsPsCGvkElDOxAUYH9axnfZb68Lbk+O497+Cbz+RRafbbLJb/XB2bPKiwszSGwWxu3JsVXvYEwdZonBJf99axd6xjfmyfc3sLPAJr/5un9u2sfWvcd4bFhHAvztz8r4NvsNdklwgD8z7kkmKMCPh95O51SxPfnNV5WdVaYuyqBDi0Z8u2eM2+EYc9UsMbgopnEoL4/tTeaBEzz9N5v85qv+sT6fzAMneGxYR/z9KrsZsTG+xRKDy67r4Jn8Nn99PrNs8pvPKS07y0uLMrimdSQ3d2vldjjGVAtLDHWAZ/JbC37zyVbSd9vkN1/ywZo9ZB86xRPDO+Jn1YKpJywx1AGeyW+9iGkcysPvrOHgCZv85guKS88ybfEOesZFMeyaFm6HY0y1scRQR0SFep78dvRUCRPfXWuT33zAe2m57Dl6midGdELEqgVTf1hiqEO6xkTx2+90599Zh3h+QYbb4ZhLKCop49UlO0hp04RBHZq7HY4x1coSQx0zpk8c4/sl8Nq/dvL55n1uh2Mu4p2VOew/dobJVi2YesgSQx30y293oWdcFE/OXc+ugyfdDsdc4FRxKTOWZXJtu2YMaNfM7XCMqXaWGOqg4AB//nhvHwL8hQf/apPf6ppZX+/m4IliJo/o6HYoxtQIrxKDiIwUke0ikikiT19iuzEioiKS4iwHisgsEdkoIltF5JkK2zYWkXkiss1ZN8BpbyoiC0Vkh/N/k6vtpC+KbRzKtLG9yThwnP+yJ7/VGceLSnj9i50M7hRNnzZN3Q7HmBpRZWIQEX9gOnAz0AUYJyJdKtkuAngUWFmh+U4gWFW7A32AB0Qk0Vk3DfhMVTsDPYGtTvvTwGJV7QAsdpYbpEEdo3liWEf+vi6fv67Y7XY4Bpj5VTZHT5XwxHCrFkz95U3FkApkqmqWqhYDc4DRlWz3HDAFKKrQpkC4iAQAoUAxcExEIoFBwJsAqlqsqkedfUYDs5yfZwG3XV6X6pdHbmzP0M4teO7jLazJOeJ2OA1a4akS/rQ8ixFdWtIjrrHb4RhTY7xJDLFAboXlPKftHBHpDcSr6scX7DsPOAnsBXKA51X1MJAEFAAzRWStiPxZRMKdfVqq6l4A5/9KZw6JyP0ikiYiaQUFBV50wzf5+Qkv3tWL1lGhPPy2TX5z05+WZ3G8qJTHrVow9Zw3iaGy7+Kdu+AtIn7AVGByJdulAmVADNAWmCwiSUAAkAzMUNXeeJLHZV0yUtU3VDVFVVOio6MvZ1efExXmmfx25FQxj862yW9uOHTiDG99tYtbe7TmmtaRbodjTI3yJjHkARWfUxgH5FdYjgC6ActEJBvoD8x3BqDH4xlHKFHVA8BXQIpzzDxVLR+PmIcnUQDsF5HWAM7/B66kY/VN15gofnNbN77eeYgXFtrkt9r2+hdZFJWU8dgwqxZM/edNYlgNdBCRtiISBIwF5pevVNVCVW2uqomqmgisAEapahqey0dDxCMcT9LYpqr7gFwR6eQcZiiwxfl5PnCf8/N9wEdX18X6486UeMalJjBj2U4W2OS3WnPgWBH/9+9sbusVS/sWjdwOx5gaV2ViUNVSYALwOZ5vDs1V1c0i8qyIjKpi9+lAI2ATngQzU1U3OOsmAu+IyAagF/A7p/33wHAR2QEMd5aN45ff7kL32Cgmz11Ptk1+qxV/XLaTkjJl0rAObodiTK2Q+vD9+JSUFE1LS3M7jFqTd+QUt77yJa0iQ/jw4YGEBvm7HVK9lX/0NIP/sIzbk2P5/R093A7HmGolIumqmnJhu8189kFxTcKYNrY32/cf52cf2uS3mvTKkuSl7IwAABRASURBVEwAJg61asE0HJYYfNQNHaN5bGhHPli7h7dX5rgdTr2Uc+gU76flMjY1ntjGoW6HY0ytscTgwyYOac+NnaJ59h+bWWuT36rdy0t24O8nPHJje7dDMaZWWWLwYX5+wtS7e9EyMoSH31nDIZv8Vm12FpzggzV5fLd/G1pGhrgdjjG1yhKDj2scFsRr9/bh0MliJs1ZR9lZG2+oDtMW7SAk0J8HB7dzOxRjap0lhnqgW2wUvxndjS8zD/Liwu1uh+Pztu87zj825PP9axNp3ijY7XCMqXWWGOqJu/rGM7ZvPNOX7mThlv1uh+PTpi7MoFFQAPcPSnI7FGNcYYmhHvnVqK50j43iibnrbPLbFdq0p5DPNu/jh9e1pXFYkNvhGOMKSwz1SEigP3+8Jxl/P+HBt9M5XVzmdkg+58WFGUSFBvKj69u6HYoxrrHEUM/ENw3jpbt7eSa//d0mv12ONTlHWLLtAPcPSiIyJNDtcIxxjSWGemhwpxZMGtqBD9bs4d1VNvnNW1MXZtAsPIjvX5vodijGuMoSQz316JAO3NAxml/P38L63KNV79DArcw6xPIdB3locDvCgwPcDscYV1liqKf8/ISX7u5FdEQwD72dzuGTxW6HVGepKi8syKBFRDD39m/jdjjGuM4SQz3WJNwz+e3gyWImzVlrk98u4svMg6zKPsyEIe0JCbQ71RpjiaGe6x4XxbOjurJ8x0FeWmRPfrtQebUQExXC3X3jq97BmAbAEkMDMDY1gbtS4nhlSSaLt9rkt4qWbj/AutyjTBzageAAqxaMAUsMDcazo7vRNSaSx99bR86hU26HUyeUVwsJTcMY0yfO7XCMqTMsMTQQIYH+vHZvH0Q8k9+KSmzy2+eb97E5/xiThnYg0N/+FIwp59Vfg4iMFJHtIpIpIk9fYrsxIqIikuIsB4rILBHZKCJbReSZCttmO+3rRCStQnsvEVlR3i4iqVfTQfMf5ZPftu47xs//vqlBT34rO6u8uDCDdtHh3NY71u1wjKlTqkwMIuIPTAduBroA40SkSyXbRQCPAisrNN8JBKtqd6AP8ICIJFZYf6Oq9rrgmaNTgF+rai/gF86yqSY3dm7BxCEdmJeex+xVuW6H45qPN+STsf8Ejw3riL+fuB2OMXWKNxVDKpCpqlmqWgzMAUZXst1zeN7Eiyq0KRAuIgFAKFAMHKvi9RSIdH6OAvK9iNFchklDOzCoYzS/mr+5QU5+Ky07y7RFO+jcKoJburd2Oxxj6hxvEkMsUPGjZZ7Tdo6I9AbiVfXjC/adB5wE9gI5wPOqethZp8ACEUkXkfsr7PMY8AcRyQWeB56hEiJyv3OpKa2goMCLbphy/n7CNGfy28PvrOFIA5v89uHaPWQdPMnjwzviZ9WCMd/gTWKo7C/n3MVpEfEDpgKTK9kuFSgDYoC2wGQRKb/J/UBVTcZzieoRERnktD8EPK6q8cDjwJuVBaWqb6hqiqqmREdHe9ENU1GT8CBm3JtMwfEzTHqv4Tz5rbj0LC8v2UH32ChGdGnpdjjG1EneJIY8oOLMnzjOv7wTAXQDlolINtAfmO8MQI8HPlPVElU9AHwFpACoar7z/wHgQzxJBOA+4APn5/crtJtq1iOuMb8e3ZUvMgqYtniH2+HUivfTc8k9fJonRnRExKoFYyrjTWJYDXQQkbYiEgSMBeaXr1TVQlVtrqqJqpoIrABGqWoanstHQ8QjHE/S2CYi4c5gNU77CGCTc8h84Abn5yFAw3jHcsnYvvGM6RPHy4t3sHTbAbfDqVFFJWW8uiST5ITGDO5oVaYxF1NlYlDVUmAC8DmwFZirqptF5FkRGVXF7tOBRnje9FcDM1V1A9AS+FJE1gOrgE9U9TNnnx8DLzjrfgfc/83DmuoiIvzmtm50aR3JY++tI/dw/Z38NmdVDnsLi5g8opNVC8ZcgtSH77KnpKRoWlpa1Ruai8o5dIpbX1lOfNMw/vbQtfXuZnKni8u4fspS2rcIZ/aP+1tiMAYQkfQLpgsANvPZOBKahTH17l5szj/GLz7aVPUOPuavK7I5eOKMVQvGeMESgzln6DUtmTikPXPT8phTj578duJMKTOW7WRQx2j6JjZ1Oxxj6jxLDOY8jw3ryPUdmvOL+ZvZmFfodjjV4i9f7eLIqRKeGN7R7VCM8QmWGMx5/P2EaWN7E90omAffTvf5yW+Fp0t444sshl3Tgl7xjd0OxxifYInBfEPT8CD+eI9n8ttjPj757c3lWRwrKuVxqxaM8ZolBlOpnvGN+eWoLvwro4CXfXTy25GTxbz1VTbf6t6KrjFRbodjjM+wxGAuanxqAnckx/Hykh0s3e57k99e/yKLk8WlPD7MqgVjLoclBnNR5ZPfOreK5LE5vjX5reD4GWZ9nc3onjF0aBnhdjjG+BRLDOaSQoP8ee3eZM6q8vA7a3zmyW8zlu2kuOwsk6xaMOayWWIwVWrTLJypd/Vi455CfjV/s9vhVGlv4WneXrmbO5Jjads83O1wjPE5lhiMV4Z1ackjN7Zjzupc5q6u209+m740E1Vl4pAObodijE+yxGC89sTwTlzXvjk//2gTm/bUzclvuYdP8d7qXO5KiSe+aZjb4RjjkywxGK95Jr/1onl4EA++nc7RU3Vv8tsrS3YgIkwY0t7tUIzxWZYYzGVp1iiY6fcks/9YEY+/t46zdWjy266DJ/nbmj3c0y+B1lGhbodjjM+yxGAuW++EJvzi211Zur2AV5Zkuh3OOdMWZRDk78dDg9u5HYoxPs0Sg7ki9/ZL4Pbesby0OINldWDy2479x/lofT7fu7YNLSJC3A7HGJ9micFcERHht9/pTqeWEXXiyW8vLdpBWKA/DwyyasGYq2WJwVwxz+S3PpSddXfy2+b8Qj7ZuJcfXdeWpuFBrsRgTH3iVWIQkZEisl1EMkXk6UtsN0ZEVERSnOVAEZklIhtFZKuIPFNh22ynfZ2IpF1wnInO620WkSlX2jlT8xKbh/OiM/nt1//Y4koMUxfuIDIkgB9dn+TK6xtT3wRUtYGI+APTgeFAHrBaROar6pYLtosAHgVWVmi+EwhW1e4iEgZsEZHZqprtrL9RVQ9ecJwbgdFAD1U9IyItrrBvppYM79KShwe344/LdpKc0Jg7U+Jr7bXX5R5l0db9PDmiI1GhgbX2usbUZ95UDKlApqpmqWoxMAfPG/eFngOmAEUV2hQIF5EAIBQoBo5V8XoPAb9X1TMAqur+yKap0uQRnRjYvhk///smNufX3uS3Fxdm0CQskO8PbFtrr2lMfedNYogFKt4DIc9pO0dEegPxqvrxBfvOA04Ce4Ec4HlVPeysU2CBiKSLyP0V9ukIXC8iK0XkXyLSt7KgROR+EUkTkbSCggIvumFqkr+f8PLY3jR1Jr8Vniqp8ddcnX2YLzIKePCGdjQKrrL4NcZ4yZvEIJW0nZvVJCJ+wFRgciXbpQJlQAzQFpgsIuUXggeqajJwM/CIiAxy2gOAJkB/4CfAXBH5Rgyq+oaqpqhqSnR0tBfdMDWtfPLbvsIiHp9b85PfXliwneaNgvnegMQafR1jGhpvEkMeUPGicRyQX2E5AugGLBORbDxv6POdAejxwGeqWuJcEvoKSAFQ1Xzn/wPAh3iSSPnrfaAeq4CzQPMr656pbckJTfjvW7uwZNsBpi+tuclvX2ceZEXWYR65sR2hQf419jrGNETeJIbVQAcRaSsiQcBYYH75SlUtVNXmqpqoqonACmCUqqbhuXw0RDzC8SSNbSIS7gxW47SPADY5h/w7MMRZ1xEIAs4boDZ123f7t+G2XjG8uCiDLzKq/zKfqvL8gu20jgphXGpCtR/fmIauysSgqqXABOBzYCswV1U3i8izIjKqit2nA43wvOmvBmaq6gagJfCliKwHVgGfqOpnzj5vAUkisgnPQPd9qlp3bshjqiQi/O727nRsEcGkOWvJO1K9k9+WZRSwJucoE4a0JyTQqgVjqpvUh/fclJQUTUtLq3pDU6t2HTzJqFe+JCk6nLkPDiA44OrfxFWVUa9+xZFTxSyZPJigAJujacyVEpF0VU25sN3+qkyNads8nOfv6sn6vOqb/LZgy3427ilk0tAOlhSMqSH2l2Vq1E1dW/HgDe14d2UO89LzrupYZ88qUxdmkNQ8nO/0jq16B2PMFbHEYGrckyM6MiCpGT/7cONVTX77ZONetu07zqRhHQjwt19dY2qK/XWZGhfg78cr43vTJCyIh95eQ+Hpy5/8VnZWeWlRBh1bNuLbPWJqIEpjTDlLDKZWNHcmv+0tPM3kK5j89tG6PewsOMnjwzri51fZnEtjTHWxxGBqTZ82Tfj5LV1YtPUAM/610+v9SsrO8tKiHXSNieSmrq1qMEJjDFhiMLXsewPaMLpXDC8s2M6XO7ybt/i39DxyDp/iieFWLRhTGywxmFolIvzP7d3p0CKCR+esJf/o6Utuf6a0jFeWZNIrvjFDOtsd2I2pDZYYTK0LCwpgxr3JFJee5aF31nCm9OJPfntvdS57jp5m8oiOVHIvRWNMDbDEYFyRFN2I5+/swfrcozz3ceWT34pKynh1SSapiU25rr3dR9GY2mKJwbhmZLfWPDAoibdX5PDBmm9Ofnt7xW4OHD9j1YIxtcwSg3HVT27qRP+kpvzXhxvZuvc/D/c7eaaUGct2cl375vRLauZihMY0PJYYjKsC/P14ZVwyUaGBnie/OZPfZv07m0Mni3liREd3AzSmAbLEYFwXHRHMH+9JZs+R00yeu57C0yW8/q8shnRuQXJCE7fDM6bBscRg6oQ+bZrys1uuYdHW/dz12r8pPF3CE8OtWjDGDZYYTJ3x/WsTGdUzhu37jzOyayu6xUa5HZIxDVKA2wEYU6588ltsk1Du6WeP7DTGLZYYTJ0SHhzAT0d2djsMYxo0ry4lichIEdkuIpki8vQlthsjIioiKc5yoIjMEpGNIrJVRJ6psG22075ORL7xXE4RedI5ls1sMsaYWlRlxSAi/sB0YDiQB6wWkfmquuWC7SKAR4GVFZrvBIJVtbuIhAFbRGS2qmY7629U1W/cSU1E4p3Xy7mCPhljjLkK3lQMqUCmqmapajEwBxhdyXbPAVOAogptCoSLSAAQChQDxyrZ90JTgaec/Y0xxtQibxJDLJBbYTnPaTtHRHoD8ar68QX7zgNOAnvxfPp/XlUPO+sUWCAi6SJyf4VjjQL2qOr6SwUlIveLSJqIpBUUFHjRDWOMMd7wZvC5spvUnPskLyJ+eD7hf7+S7VKBMiAGaAIsF5FFqpoFDFTVfBFpASwUkW1AGvAzYERVQanqG8AbACkpKVZZGGNMNfGmYsgD4issxwH5FZYjgG7AMhHJBvoD850B6PHAZ6paoqoHgK+AFABVzXf+PwB8iCeJtAPaAuudY8UBa0TEHttljDG1xJvEsBroICJtRSQIGAvML1+pqoWq2lxVE1U1EVgBjFLVNDyXj4aIRziepLFNRMKdwWqc9hHAJlXdqKotKhwrD0hW1X3V12VjjDGXUmViUNVSYALwObAVmKuqm0XkWWc84FKmA42ATXgSzExV3QC0BL4UkfXAKuATVf3sKvphjDGmmoiq71+eF5ECYPcV7t4c8O7hw3Wf9aXuqS/9AOtLXXU1fWmjqtEXNtaLxHA1RCRNVVPcjqM6WF/qnvrSD7C+1FU10Re7iZ4xxpjzWGIwxhhzHksMzlyIesL6UvfUl36A9aWuqva+NPgxBmOMMeezisEYY8x5LDEYY4w5T4NLDCLiLyJrReRjZ7mtiKwUkR0i8p4zu9snVNKXv4jILucZF+tEpJfbMXqjsmdziEhTEVnonJeFItLE7Ti9cZG+/EpE9lQ4L99yO05viEhjEZknItuc56kM8MXzcpF++Nw5EZFOFeJdJyLHROSxmjgnDS4xAJPwzOAu97/AVFXtABwBfuRKVFfmwr4A/ERVezn/1rkR1BW60Ym5/PvYTwOLnfOy2Fn2FRf2BTy/Y+Xn5VPXIrs80/Dc66wz0BPP75ovnpfK+gE+dk5UdXt5vEAf4BSe+8xV+zlpUIlBROKAW4A/O8sCDMFze3CAWcBt7kR3eS7sSz00Gs/5AB86L/WFiEQCg4A3AVS1WFWP4mPn5RL98HVDgZ2qupsaOCcNKjEAL+F5ANBZZ7kZcNS5HxRU8qyJOuzCvpT7rYhsEJGpIhLsQlxXorJnc7RU1b0Azv8tXIvu8lT6nBFggnNe3vKFyy9AElAAzHQuV/7ZueGlr52Xi/UDfO+cVDQWmO38XO3npMEkBhG5FTigqukVmyvZtM5/f/cifQF4BugM9AWaAj+t7diu0EBVTQZuBh4RkUFuB3QVKuvLDDy3lO+F56FVL7gYn7cCgGRghqr2xvPALV+4bHShi/XDF88JAM446Cjg/Zp6jQaTGICBwCjnOQ9z8FxCegloLJ5Hj8I3nzVRV32jLyLytqruVY8zwEw8z7io8y7ybI79ItIawPn/gHsReq+yvqjqflUtU9WzwJ/wjfOSB+Spavkz3OfheYP1tfNSaT989JyUuxlYo6r7neVqPycNJjGo6jOqGuc852EssERV7wGWAmOcze4DPnIpRK9dpC/3VvjlEDzXGTe5GKZXLvZsDjzP/LjP2cwnzsvF+lJ+XhzfwQfOi/MMlFwR6eQ0DQW24GPn5WL98MVzUsE4/nMZCWrgnHjzaM/67qfAHBH5DbAWZ5DKR70jItF4LpGtAx50OR5vtAQ+9OQyAoB3VfUzEVkNzBWRH+F54NOdLsborYv15a/OV4cVyAYecC/EyzIRz+9UEJAF/ADPh0lfOy+V9eNlXzwnIhIGDOf8eH9PNZ8TuyWGMcaY8zSYS0nGGGO8Y4nBGGPMeSwxGGOMOY8lBmOMMeexxGCMMeY8lhiMMcacxxKDMcaY8/x/FlO0TW40YfsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### min_child_samples=50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 行采样参数 sub_samples/bagging_fraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed: 10.0min remaining:  2.5min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 12.9min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=50,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=1.0,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'subsample': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':50,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "subsample_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( subsample = subsample_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.485439275836877\n",
      "{'subsample': 0.7}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9b3/8dcnKwTCGgQhwQRlCxBBAgH1qnXFDUSwRVwQoaIt+rtK61KtraAVrK2t97pU2RRxRVRE1FpvqRsEgsgW9oDIvhMkZJt8f39koIEGMiHLmcm8n49HHsycJfM+x/G8c86cOcecc4iISPiJ8DqAiIh4QwUgIhKmVAAiImFKBSAiEqZUACIiYSrK6wCVkZCQ4JKTk72OISISMhYtWrTbOdeivHEhVQDJyclkZWV5HUNEJGSY2fcnGqdDQCIiYUoFICISplQAIiJhKqACMLN+ZrbazNaZ2YMnmW6wmTkzS/c/jzGzKWa2zMyWmNlFZabt6R++zsyeNTOr8tKIiEjAKiwAM4sEngOuBFKBG80stZzp4oF7gMwyg38O4JzrBlwG/MnMjrzmC8AdQHv/T79TXwwREamsQPYAegPrnHM5zrlC4E1gQDnTjQOeAvLLDEsFPgdwzu0E9gPpZnY60Mg5N8+VXo3uVeC6U18MERGprEAKoA3wQ5nnm/3DjjKzHkCSc272cfMuAQaYWZSZpQA9gST//JtP9jvL/O47zCzLzLJ27doVQFwREQlEIAVQ3rH5o9eQ9h/SeQYYU850kynduGcBfwG+AYor+p3HDHTuJedcunMuvUWLcr/LcFLFvhJemLuexZv2VXpeEZG6LJAC2EzpX+1HJAJbyzyPB7oCc81sI9AHmGVm6c65Yufcvc657s65AUATYK3/dyae5HdWm/ziEl6dt5EH3l1KYXFJTbyEiEhICqQAFgLtzSzFzGKAIcCsIyOdcweccwnOuWTnXDIwH+jvnMsyszgzawBgZpcBxc65bOfcNuCgmfXxn/1zK/BBNS8bAA1jo3hiYFfW7PiR5+euq4mXEBEJSRUWgHOuGBgNfAqsBN52zq0ws7Fm1r+C2U8DvjWzlcADwC1lxt0FTATWAeuBj08hf0Au7tSSAd1b89w/17Fmx8GaehkRkZBioXRLyPT0dHeq1wLa82MBl/75XyQnNGDGnecSGaGvHYhI3Wdmi5xz6eWNC5tvAjdvGMvvru3C4k37eXXeRq/jiIh4LmwKAGBA99Zc1LEFf/x0NZv35XkdR0TEU2FVAGbGEwO7YcBv3ltOKB3+EhGpbmFVAABtmtTn/n6d+GLNLt5bvMXrOCIingm7AgC4pc8ZpJ/RlLGzs9n9Y4HXcUREPBGWBRARYYwflEZegY/fz1rhdRwREU+EZQEAnHVaQ+6++CxmL93GP7J3eB1HRKTWhW0BAIy68Ew6tYrnkfeXk5tf5HUcEZFaFdYFEBMVwYRBaew8mM+Ej1d5HUdEpFaFdQEAnJ3UhNvPS2F65iYyc/Z4HUdEpNaEfQEA3Hd5B5Ka1efBmcvIL/J5HUdEpFaoAIC4mCieHJjGht2H+Ovna72OIyJSK1QAfue3T+Cn6Ym89EUOy7cc8DqOiEiNUwGU8fBVqTRrEMODM5dS7NPNY0SkblMBlNE4Lpqx/buwfEsuE7/a4HUcEZEapQI4zpXdTueKLi155rM1bNh9yOs4IiI1RgVQjrEDuhITFcGD7y6lpERXDBWRukkFUI6Wjerx8FWdydywlzcX/uB1HBGRGqECOIGf9Uqib7vmPDlnJdsP5HsdR0Sk2qkATsDMePL6bhSVlPDbD3TzGBGpe1QAJ5Gc0ID7LuvAZ9k7mLNsu9dxRESqlQqgArefl0K3No353azl7M8r9DqOiEi1UQFUICqy9Iqh+/OKePyjlV7HERGpNiqAAKS2bsSoC9sxY9Fmvly7y+s4IiLVQgUQoLsvbk+7Fg14aOYyDhUUex1HRKTKVAABqhcdyfjr09i87zB/+vsar+OIiFSZCqASeqc045Y+ZzDlmw0s3rTP6zgiIlWiAqik+/t1pFWjejz47jIKi3XFUBEJXSqASoqvF80TA7uyesdBXpi73us4IiKnTAVwCi7u1JL+Z7fmf/+5lrU7DnodR0TklKgATtHvrk2lYWwU97+7FJ+uGCoiIUgFcIqaN4zl0WtTWbxpP6/O2+h1HBGRSlMBVMF13dtwUccW/PHT1Wzel+d1HBGRSlEBVIGZ8fh1XTHgN+/piqEiElpUAFWU2DSO+/t14os1u3hv8Rav44iIBCygAjCzfma22szWmdmDJ5lusJk5M0v3P482s1fMbJmZrTSzh8pM+//MbLmZrTCz/676onjnlj5n0POMpoydnc3uHwu8jiMiEpAKC8DMIoHngCuBVOBGM0stZ7p44B4gs8zgG4BY51w3oCcwysySzawr8HOgN3A2cI2Zta/qwnglIsKYMKgbeQU+Hvsw2+s4IiIBCWQPoDewzjmX45wrBN4EBpQz3TjgKaDs/RMd0MDMooD6QCGQC3QG5jvn8pxzxcC/gIGnvhjeO+u0eEZffBYfLtnKP7J3eB1HRKRCgRRAG6DsndE3+4cdZWY9gCTn3Ozj5p0BHAK2AZuAp51ze4HlwAVm1tzM4oCrgKTyXtzM7jCzLDPL2rUruC/FfOeFZ9KxZTyPvL+c3Pwir+OIiJxUIAVg5Qw7erqLmUUAzwBjypmuN+ADWgMpwBgza+ecWwlMAD4DPgGWAOVeY9k595JzLt05l96iRYsA4nonJiqCCYPT2Hkwnwkfr/I6jojISQVSAJs59q/zRGBrmefxQFdgrpltBPoAs/wfBA8FPnHOFTnndgJfA+kAzrlJzrlznHMXAHuBtVVdmGDQPakJw89LYXrmJjJz9ngdR0TkhAIpgIVAezNLMbMYYAgw68hI59wB51yCcy7ZOZcMzAf6O+eyKD3sc7GVakBpOawCMLPT/P+2Ba4H3qjG5fLUmMs7kNSsPg/NXEZ+kc/rOCIi5aqwAPwf0o4GPgVWAm8751aY2Vgz61/B7M8BDSk95r8QmOKcW+of966ZZQMfAr90ztWZC+zHxUTx5MA0cnYf4tnP68SOjYjUQRZK315NT093WVlZXscI2K/fWcLMxVuYNfo8urRu7HUcEQlDZrbIOZde3jh9E7gGPXJ1Kk3jYnjg3aUU+3TzGBEJLiqAGtQ4LpqxA7qwfEsuE7/a4HUcEZFjqABq2JVdW3F5akue+WwNG3Yf8jqOiMhRKoAaZmaMu64rMVERPDRzqa4YKiJBQwVQC1o2qsfDV3Vmfs5e3lz4Q8UziIjUAhVALflZryT6tmvOH+asZEdufsUziIjUMBVALTEznry+G4XFJTzyvm4eIyLeUwHUouSEBtx3WQc+y97BnGXbvY4jImFOBVDLRpyfQrc2jfndrOXszyv0Oo6IhDEVQC2Lioxg/KBu7Msr4vGPVnodR0TCmArAA11aN2bUBe2YsWgzX64N7nsciEjdpQLwyD2XtKddQgMemrmMvMJyb4UgIlKjVAAeqRcdyfhBaWzed5g//X2N13FEJAypADzUO6UZN/dpy+SvN7B4U525GraIhAgVgMce6NeJVo3q8eC7yygs1hVDRaT2qAA8Fl8vmsev68rqHQd5Ye56r+OISBhRAQSBSzq35NqzW/O//1zL2h0HvY4jImFCBRAkfndtKg1io3jg3aX4SnSZCBGpeSqAIJHQMJZHr0nl2037mTZvo9dxRCQMqACCyMAebbiwQwue+nQ1m/fleR1HROo4FUAQMTOeGNgVgN+8pyuGikjNUgEEmcSmcdx/RUe+WLOL9xZv8TqOiNRhKoAgdEvfZM5p24Sxs7PZ/WOB13FEpI5SAQShyAhjwqA08gp8PPZhttdxRKSOUgEEqfYt4/nlT87iwyVb+XzlDq/jiEgdpAIIYndddCYdW8bzyPvLOZhf5HUcEaljVABBLCYqggmD09iRm8+ET1Z5HUdE6hgVQJDrntSE4eel8Nr8TWTm7PE6jojUISqAEDDm8g4kNavPQzOXkV/k8zqOiNQRKoAQEBcTxZMD08jZfYhnP1/rdRwRqSNUACHi/PYJDO6ZyN++yGHF1gNexxGROkAFEEIeubozTeNieODdpRT7dPMYEakaFUAIaRIXw2P9u7B8Sy6TvtrgdRwRCXEqgBBzVbdWXJ7akj9/toYNuw95HUdEQpgKIMSYGeOu60pMVAQPzVyqK4aKyCkLqADMrJ+ZrTazdWb24EmmG2xmzszS/c+jzewVM1tmZivN7KEy095rZivMbLmZvWFm9aq+OOGhZaN6/OaqzszP2cubC3/wOo6IhKgKC8DMIoHngCuBVOBGM0stZ7p44B4gs8zgG4BY51w3oCcwysySzayNf9p051xXIBIYUtWFCSdDeiXRp10z/jBnJTty872OIyIhKJA9gN7AOudcjnOuEHgTGFDOdOOAp4CyWyMHNDCzKKA+UAjk+sdFAfX94+KArae2COHJzBh/fRqFxSX89n3dPEZEKi+QAmgDlD3OsNk/7Cgz6wEkOedmHzfvDOAQsA3YBDztnNvrnNsCPO0ftg044Jz7e3kvbmZ3mFmWmWXt2rUrkGUKG8kJDbj3sg78PXsHHy/f7nUcEQkxgRSAlTPs6J+bZhYBPAOMKWe63oAPaA2kAGPMrJ2ZNaV0LyLFP66Bmd1c3os7515yzqU759JbtGgRQNzwMvL8FLq2acSjH6xgf16h13FEJIQEUgCbgaQyzxM59nBNPNAVmGtmG4E+wCz/B8FDgU+cc0XOuZ3A10A6cCmwwTm3yzlXBMwEzq3qwoSjqMgIJgxKY19eIY9/tNLrOCISQgIpgIVAezNLMbMYSj+snXVkpHPugHMuwTmX7JxLBuYD/Z1zWZQe4rnYSjWgtBxW+Yf3MbM4MzPgEkBbr1PUpXVjRl3QjhmLNvPlWh0mE5HAVFgAzrliYDTwKaUb6bedcyvMbKyZ9a9g9ueAhsBySotkinNuqXMuk9LPB74FlvlzvHTqiyH3XNKedgkNeGjmMvIKi72OIyIhwELp7JH09HSXlZXldYyglZmzh5+9NJ8R56fw22v+40xdEQlDZrbIOZde3jh9E7gOyWjXnJsy2jLl6w1898N+r+OISJBTAdQxD17ZidPi6/HAjKUUFuuKoSJyYiqAOia+XjSPX9eV1TsO8uK/1nsdR0SCmAqgDro0tSXXpJ3O//zfWtbuOOh1HBEJUiqAOur3/bvQIDaKB95diq8kdD7oF5HaowKooxIaxvLoNal8u2k/0+Zt9DqOiAQhFUAdNrBHGy7o0IKnPl3N5n15XscRkSCjAqjDzIw/DOwKwMPv6YqhInIsFUAdl9g0jl9f0ZF/rdnF+99t8TqOiAQRFUAYuLVvMj3aNmHsh9ns/rHA6zgiEiRUAGEgMsKYMCiNHwuKeezDbK/jiEiQUAGEiQ4t4xn9k/Z8uGQrn6/c4XUcEQkCKoAwctdFZ9KxZTyPvL+cg/lFXscREY+pAMJITFQE4wd1Y3tuPhM+WeV1HBHxmAogzPRo25Th56bw2vxNLNiw1+s4IuIhFUAY+tUVHUhsWp8H311KfpHP6zgi4hEVQBiKi4niDwO7kbP7EM9+vtbrOCLiERVAmLqgQwsGnZPI377IYcXWA17HEREPqADC2G+v6UzTuGgeeHcpxT7dPEYk3KgAwliTuBge69+V5VtymfTVBq/jiEgtUwGEuau6teKy1Jb8+bM1bNx9yOs4IlKLVABhzswYN6ArMZERPDRzma4YKhJGVABCq8b1eOiqzszL2cNbC3/wOo6I1BIVgAAwpFcSGSnNeGLOSnbk5nsdR0RqgQpAAIiIMMYPSqOwuITfvq+bx4iEAxWAHJWS0IB7L+vA37N3MHvpNq/jiEgNUwHIMUaen8LZSU341TtLmLd+j9dxRKQGqQDkGFGREUwelk7bZnGMeGUh327a53UkEakhKgD5D80bxvLayAxaxMdy2+QFulSESB2lApBytWxUj+kjM2gYG8UtkxawbudBryOJSDVTAcgJJTaN47WRGUSYMfTlTL7fo28Ki9QlKgA5qXYtGjJ9ZAaFvhKGvpzJ1v2HvY4kItVEBSAV6tgqnmm3Z5B7uIibJ2ay62CB15FEpBqoACQg3RIbM2V4L7YdyOfmiZnsO1TodSQRqaKACsDM+pnZajNbZ2YPnmS6wWbmzCzd/zzazF4xs2VmttLMHvIP72hm35X5yTWz/66eRZKakp7cjInD0tmw5xDDpizgYH6R15FEwoKvpGa+mV9hAZhZJPAccCWQCtxoZqnlTBcP3ANklhl8AxDrnOsG9ARGmVmyc261c667c667f3ge8F6Vl0Zq3HlnJfD80HPI3prL7VMXkldY7HUkkTrLOcff/rWe26YsoLC4+m/aFMgeQG9gnXMuxzlXCLwJDChnunHAU0DZK4k5oIGZRQH1gUIg97j5LgHWO+e+r2x48calqS35y5DuLPp+H6OmLdKN5UVqgK/E8diH2Tz58Soa14/GUf17AYEUQBug7DWCN/uHHWVmPYAk59zs4+adARwCtgGbgKedc3uPm2YI8EZlQov3rklrzYRBaXy5djejX19MkW4pKVJt8ot8jH79W6Z+s5GR56fw7JAexEZFVvvrBFIAVs6wo1VkZhHAM8CYcqbrDfiA1kAKMMbM2pWZNwboD7xzwhc3u8PMsswsa9euXQHEldpyQ3oSYwd04R8rd3DvW9/V2HFKkXCyP6+QWyct4OPl23nk6s48ck0qERHlbYarLiqAaTYDSWWeJwJbyzyPB7oCc80MoBUwy8z6A0OBT5xzRcBOM/saSAdy/PNeCXzrnNtxohd3zr0EvASQnp6uLUyQubVvMnmFPsZ/vIr60ZFMGJRWY29Wkbpuy/7DDJu8gE178vifG3tw7dmta/T1AtkDWAi0N7MU/1/sQ4BZR0Y65w445xKcc8nOuWRgPtDfOZdF6WGfi61UA6APsKrM774RHf4JeXdeeCb3XNKedxZtZuzsbN1LQOQUrNyWy/XPf82O3Hxeub13jW/8IYA9AOdcsZmNBj4FIoHJzrkVZjYWyHLOzTrJ7M8BU4DllB5KmuKcWwpgZnHAZcCoKi6DBIF7L21PXkExE7/aQP2YSO6/oiP+PUIRqcA363YzatoiGsRGMePOc+nYKr5WXjeQQ0A45+YAc44b9ugJpr2ozOMfKT0VtLzp8oDmgQaV4GZmPHx1Zw4X+Xhh7noaxEQy+uL2XscSCXoffLeFX72zhJSEBkwd3pvWTerX2msHVAAigTAzxg3oyuFCH0//fQ31Y6IYcX6K17FEgpJzjpe/zOEPc1aRkdKMl25Np3H96FrNoAKQahURYTw1OI3DRT7Gzc6mfnQkQzPaeh1LJKj4ShzjZmcz9ZuNXJ12On/+6dk1cppnRVQAUu2iIiP465AeHJ6WxcPvLyMuJpLrerSpeEaRMJBf5OO+t79jzrLtjDg/hYev6uzZmXO6GJzUiJioCF68uScZKc0Y884SPlm+3etIIp47kFfErZMXMGdZ6Tn+v63Bc/wDoQKQGlMvOpKJw3qRltiYu9/4lrmrd3odScQzW/cfZvCL3/Ddpv08e2MPRv5Xu4pnqmEqAKlRDWOjmDq8N+1Pi2fUtEXMz9njdSSRWrdyWy4Dn/+a7QfymXp7L/rXwjn+gVABSI1rXD+aaSN6k9QsjhFTF/Ltpn1eRxKpNd+s381PX5yHYbxzV1/OPTPB60hHqQCkVjRvGMv0kRkkxMdy2+QFrNh6wOtIIjVu1pKtDJu8gNOb1GPmL86lU6tGXkc6hgpAak3LRvWYPjKDhrFR3DppAet2HvQ6kkiNmfhlDve8sZgebZvyzqhza/ULXoFSAUitSmwax2sjMzAzbpqYyfd7DnkdSaRalZQ4xn6YzeMfreTqbqfz6u29aRxXu1/wCpQKQGpduxYNmT4yg4LiEoa+nMnW/Ye9jiRSLfKLfNz95mImf72B4ecl8z839qBedO1/wStQKgDxRMdW8Uy7PYPcw0XcPDGTXQcLvI4kUiVHzvH/aOk2Hr6qM496fI5/IFQA4pluiY2ZMrwX2w7kc8ukTPYdKvQ6ksgp2br/MDf87RsWb9rHX4d05+cXtAuJq+GqAMRT6cnNmDgsnZzdhxg2ZQEH84u8jiRSKau253L989+wbX8+rwzvzYDuoXPZExWAeO68sxJ4fug5ZG/N5fapC8krLPY6kkhA5q3fww0vzsPhePvOvpx7VvCc4x8IFYAEhUtTW/KXId1Z9P0+Rk1bRH6Rz+tIIif1of8c/1aN6jHzF+fR+fTgOsc/ECoACRrXpLVmwqA0vly7m9GvL6bIV+J1JJFyTfwyh7vfWEz3pCa8c2df2gThOf6BUAFIULkhPYmxA7rwj5U7uO/tJfhKdH9hCR4l/uv4P/7RSq7q1opXR/SmSVyM17FOme4HIEHn1r7J5BX6GP/xKupHRzD++rSgP51O6r6CYh/3vb2Ej5Zu47Zzk/ntNalEhvj7UgUgQenOC88kr9DHs5+vJS4mit9dmxoSp9VJ3XTgcBGjpmUxP2cvv7mqEz//r9A4zbMiKgAJWvde2p68gmImfrWBuJhI7u/XyetIEoa2HTjMbZMXkrP7R/46pHtIneZZERWABC0z4+GrO5NX5OP5ueuJi4lk9MXtvY4lYWT19oPcNmUBB/OLmTq8N+eF2GmeFVEBSFAzMx4f0JX8Qh9P/30N9WOiGHF+itexJAzMz9nDz1/Non50JG+P6ktq69A7zbMiKgAJehERxlOD0zhc5GPc7GziYiK5sXdbr2NJHTZ76Vbue2sJbZvHMXV4LxKbxnkdqUboNFAJCVGREfx1SA8u6tiC37y3jPcXb/E6ktRRk77awN1vLObspMbMuLNvnd34gwpAQkhMVAQv3tyTjJRmjHlnCZ8s3+51JKlDSkocT3yUzbjZ2VyR2oppIzJC+hz/QKgAJKTUi45k4rBepCU25u43vmXu6p1eR5I6oKDYx/976zte/nIDw/qewXM3nRPU1/GvLioACTkNY6OYOrw37U+LZ9S0RczP2eN1JAlhBw4XMWzyAj5cspUHr+zE7/t3CfkveAVKBSAhqXH9aKaN6E1SszhGTF3I4k37vI4kIWj7gXx+9rd5LPp+H3/5WXfuvPDMOvEFr0CpACRkNW8Yy/SRGSTExzJs8gKyt+Z6HUlCyJodB7n++a/ZvO8wU27rzXU96s4XvAKlApCQ1rJRPaaPzKBhbBS3TMpk3c6DXkeSEJCZs4fBL3xDcYnjrVF9OL993fqCV6BUABLyEpvG8drIDMyMmyZmsmlPnteRJIh9tHQbt0xaQIv4WGb+4ly6tG7sdSTPqACkTmjXoiHTR2ZQUFzC0Inz2XbgsNeRJAhN/moDo9/4lrTExrx717l1+hz/QKgApM7o2CqeabdncCCviJtezmTXwQKvI0mQKClx/GHOSsbOzuby1Ja8NrLun+MfCBWA1CndEhszeXgvth3I55ZJmezPK/Q6knisoNjHf7/1HS99kcOtfc/g+Zt6hsU5/oEIqADMrJ+ZrTazdWb24EmmG2xmzszS/c+jzewVM1tmZivN7KEy0zYxsxlmtso/rm/VF0cEeiU34+Vb08nZfYhhkxdwML/I60jikdz8IoZPWcisJVt5oF8nHgujc/wDUWEBmFkk8BxwJZAK3GhmqeVMFw/cA2SWGXwDEOuc6wb0BEaZWbJ/3F+BT5xznYCzgZWnvhgixzq/fQLPDz2HFVtzGTE1i7zCYq8jSS3bfiCfn744jwUb9vLnn57NXReF1zn+gQhkD6A3sM45l+OcKwTeBAaUM9044Ckgv8wwBzQwsyigPlAI5JpZI+ACYBKAc67QObf/1BdD5D9dmtqSvwzpTtb3exk1bREFxT6vI0ktWes/x/+HvXlMGd6L689J9DpSUAqkANoAP5R5vtk/7Cgz6wEkOedmHzfvDOAQsA3YBDztnNsLtAN2AVPMbLGZTTSzBuW9uJndYWZZZpa1a9eugBZK5Ihr0lozYVAaX67dzejXF1PkK/E6ktSwBRv2MuiFbygqcbw1qi//1b6F15GCViAFUN4+kzs60iwCeAYYU850vQEf0BpIAcaYWTtK70NwDvCCc64HpSVR7mcLzrmXnHPpzrn0Fi30H1Iq74b0JMYO6MJn2Tu47+0l+EpcxTNJSPp42TZunpRJQnwsM+86l65twvcc/0AEckOYzUBSmeeJwNYyz+OBrsBc//G1VsAsM+sPDKX0OH8RsNPMvgbSgS+Azc65I58XzOAEBSBSHW7tm0xeoY/xH6+ifnQE469PI0IfBtYpU7/ewGOzszmnbVMm3ppO0wY6zbMigewBLATam1mKmcUAQ4BZR0Y65w445xKcc8nOuWRgPtDfOZdF6WGfi61UA6APsMo5tx34wcw6+n/NJUB29S2WyH+688IzueeS9rydtZmxs7NxTnsCdUFJiePJOSv5/YfZXNa5JdNHZmjjH6AK9wCcc8VmNhr4FIgEJjvnVpjZWCDLOTfrJLM/B0wBllN6KGmKc26pf9zdwHR/qeQAw6uwHCIBuffS9uQVFDPxqw3ExURyf79OXkeSKigsLuHXM5bwwXdbuaXPGWF1KefqENA9gZ1zc4A5xw179ATTXlTm8Y+Ungpa3nTfUXo4SKTWmBkPX92ZvCIfz89dT4PYKH75k7O8jiWnIDe/iLteW8TX6/bw6ys68gud5llpuim8hB0z4/EBXckv9PHHT1dTLzqSEeeneB1LKmFHbj7DJi9g3c4f+dMNZzOop07zPBUqAAlLERHGU4PTyCv0MW52NnExkdzYu63XsSQA63YeZNjkhezPK2Tybb24oIPODjxVuhaQhK2oyAievbEHF3VswW/eW8YH323xOpJUYOHGvQx6YR6FvhLeGtVXG/8qUgFIWIuJiuDFm3uSkdKM+95ewifLt3sdSU7gk+XbuGliJs0bxOgc/2qiApCwVy86konDepGW2Ji73/iWuat3eh1JjvPKNxu5a/q3dG3diBl3nUtSs/C+jn91UQGIAA1jo5g6vDftT4tn1LRFzM/Z43UkofQc//Efr+J3s1ZwaeeWTB/Zh2DTj4wAAAgCSURBVGY6x7/aqABE/BrXj2baiN4kNYtjxNSFLN60z+tIYa2wuIQx7yzhxX+t5+Y+bXnx5p7Uj9F1/KuTCkCkjOYNY5k+MoOE+FiGTV5A9tZcryOFpYP5Rdw+dSHvLd7Cr6/oyLgBXfUFrxqgAhA5TstG9Zg+MoOGsVHcMimTdTt/9DpSWNmRm89P/zaf+Tl7ePqGs/nlT87SF7xqiApApByJTeN4bWQGZsZNE+ezaU+e15HCwrqdB7n++W/4fs8hJt3Wi8H6gleNUgGInEC7Fg2ZPjKDguIShk6cz7YDh72OVKdl+c/xLygu4a07+nKhzvGvcSoAkZPo2CqeabdncCCviJtezmTXwQKvI9VJnyzffvQc//d+cS7dEnWOf21QAYhUoFtiYyYP78W2A/ncMimT/XmFXkeqU16dt5G7pi8iVef41zoVgEgAeiU34+Vb08nZfYhhkxdwML/I60ghzznHhE9W8egHK7ikU0te1zn+tU4FIBKg89sn8PzQc1ixNZcRU7M4XKibzJ+qwuISxry9hBfmrmdoRltevPkcnePvARWASCVcmtqSZ37Wnazv93LHtCwKilUCgXDOUVDsIze/iK37DzPilYXMXLyFX13egSeu60pUpDZFXtDloEUq6dqzW5Nf5OPXM5Yy+vXFPH/TOUQH8Qas2FdCQXEJhcVl//VR4H9+5PGR8QVFPgp9JRQUlRz9t6DY9+/xZR6XN6y81ygsLjkmU2SE8cfBadyQnnSC1FIbVAAip+CG9CQOF/l49IMVjHl7Cc/8rPt/fFO19K/e/9wgnnBDW+w7yUb3RBta/0a2zO/49+8vHeYrqfq9j6MijJioCGKjIoiNivz34+gIYiJLh8XFRfmHRZYOiy6dJsY/T+zR+SM4O6kJaYlNqpxLqkYFIHKKbu2bTF6hj/Efr2Lhxr0YHPuXtq+kwt8RiPI2ojH+DW1sZAQNY6No3uDf05xo4xtz/Ma77Hj/hrxe9LHTxESVDtchmrpJBSBSBXdeeCbx9aKYn7O3zAb1Pze0scdtaEv/jTzJhrr0eXSk6TIIUmNUACJVdFPGGdyUcYbXMUQqTft1IiJhSgUgIhKmVAAiImFKBSAiEqZUACIiYUoFICISplQAIiJhSgUgIhKmzLmqXyektpjZLuD7U5w9AdhdjXGqi3JVjnJVjnJVTl3MdYZzrtz7a4ZUAVSFmWU559K9znE85aoc5aoc5aqccMulQ0AiImFKBSAiEqbCqQBe8jrACShX5ShX5ShX5YRVrrD5DEBERI4VTnsAIiJShgpARCRMhXwBmFk/M1ttZuvM7MFyxt9mZrvM7Dv/z8gy44aZ2Vr/z7AgyuUrM3xWbebyT/NTM8s2sxVm9nqZ4Z6trwpy1dj6CiSbmT1T5vXXmNn+MuO8fI+dLJdn7zEza2tm/zSzxWa21MyuKjPuIf98q83simDIZWbJZna4zPp6sZZznWFmn/szzTWzxDLjqvb+cs6F7A8QCawH2gExwBIg9bhpbgP+t5x5mwE5/n+b+h839TqXf9yPHq6v9sDiI+sCOC1I1le5uWpyfQWa7bjp7wYmB8M6O1GuIHiPvQTc5X+cCmws83gJEAuk+H9PZBDkSgaWe7i+3gGG+R9fDEyrrvdXqO8B9AbWOedynHOFwJvAgADnvQL4zDm31zm3D/gM6BcEuWpSILl+DjznXyc453b6h3u9vk6Uq6ZV9r/ljcAb/sder7MT5apJgeRyQCP/48bAVv/jAcCbzrkC59wGYJ3/93mdqyYFkisV+Nz/+J9lxlf5/RXqBdAG+KHM883+Yccb5N99mmFmSZWct7ZzAdQzsywzm29m11VTpkBzdQA6mNnX/tfvV4l5vcgFNbe+As0GlO6qU/qX6/9Vdt5azgXevsd+D9xsZpuBOZTunQQ6rxe5AFL8h4b+ZWb/VU2ZAs21BBjkfzwQiDez5gHOe1KhXgBWzrDjz2v9EEh2zqUB/wBeqcS8XuQCaOtKv/Y9FPiLmZ1Zi7miKD3cchGlfzVONLMmAc7rRS6oufUVaLYjhgAznHO+U5i3sqqSC7x9j90ITHXOJQJXAdPMLCLAeb3ItY3S9dUDuA943cwaUT0CyfUr4EIzWwxcCGwBigOc96RCvQA2A2X/ck7kuN0259we51yB/+nLQM9A5/UoF865rf5/c4C5QI/ayuWf5gPnXJF/N3w1pRteT9fXSXLV5PoKNNsRQzj2MIvX6+xEubx+j40A3va//jygHqUXO/N6fZWby39Iao9/+CJKj9l3qK1czrmtzrnr/QX0sH/YgQCX6eRq4oON2vqh9K/CHEp3b498gNLluGlOL/N4IDDf/fsDlA2UfnjS1P+4WRDkagrE+h8nAGs5yYd7NZCrH/BKmdf/AWgeBOvrRLlqbH0Fms0/XUdgI/4vVwbDe+wkubx+j30M3OZ/3JnSjZYBXTj2Q+Acqu9D4KrkanEkB6Uf1m6p5fd+AhDhf/wEMLa63l/V8j+Jlz+U7qqtobSVH/YPGwv09z9+EljhX7H/BDqVmfd2Sj9oWgcMD4ZcwLnAMv/wZcCIWs5lwJ+BbP/rDwmS9VVurppeX4Fk8z//PTC+nHk9W2cnyhUE77FU4Gv/638HXF5m3of9860GrgyGXJQefz/y/+q3wLW1nGswpSW9BpiIv7yr4/2lS0GIiISpUP8MQERETpEKQEQkTKkARETClApARCRMqQBERMKUCkBEJEypAEREwtT/B69qYm0ZnSb3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "\n",
    "x_axis = subsample_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.4899558 , -0.48726071, -0.48543928, -0.4854911 , -0.48703753])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### subsample=0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 列采样参数 sub_feature/feature_fraction/colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  9.9min remaining:  2.5min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 12.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=1.0,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=50,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=394,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.7,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':50,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.485439275836877\n",
      "{'colsample_bytree': 0.7}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXwV5fX48c+BEJYQ1oQ9kATCEkC2iIDI6oLKYhEtai0uFaVFrdW2Wqsg1LrX+rOopQjS1h21RlQQFxTZA8iSECCEkIQ1bGEN2c7vjzvhe8FIbsgy9ybn/Xr5yp2Z55k5M1zvuTPnzjyiqhhjjDFFargdgDHGGP9iicEYY8xZLDEYY4w5iyUGY4wxZ7HEYIwx5ixBbgdQHsLCwjQyMtLtMIwxJqCsWbPmgKqGnzu/SiSGyMhIEhIS3A7DGGMCiojsLG6+XUoyxhhzFksMxhhjzmKJwRhjzFksMRhjjDmLJQZjjDFnscRgjDHmLJYYjDHGnMUSgzHGBKCcvAKmxieyNzun3NdticEYYwLQzO9SeWNZGqkHjpf7ui0xGGNMgMk8fJJXFqdwbfeWDGgfVu7rt8RgjDEB5qnPkgH407VdKmT9lhiMMSaALEs5wKcb9/DrIR1o3ahuhWzDEoMxxgSI/IJCpn6SSESTukwcFF1h27HEYIwxAeI/K3aydd9x/nxtLHVq1ayw7VhiMMaYAHDw+Gn+tmgrl8WEcWVs8wrdliUGY4wJAM8t3MKp3AKmjOqKiFTotiwxGGOMn9uQeYR3EzK4/dJIOjSrX+Hbs8RgjDF+rLBQmRqfSNOQ2tw3PKZStmmJwRhj/NhH63axNv0IfxzRidA6tSplmz4lBhEZISJbRCRFRB4+T7txIqIiEudM1xKRuSKyUUQ2i8gjXm0bicg8EUl2lvV35vcUkRUi8oOIJIhI37LupDHGBKJjOXk8vSCZnhGNuL53m0rbbomJQURqAjOAq4FY4CYRiS2mXShwH7DSa/YNQG1V7Q70Ae4WkUhn2UvAAlXtDPQANjvznwWeUNWewOPOtDHGVDsvf53CgeOneWJ0V2rUqNiCszdfzhj6AimqmqqqucA7wJhi2k3H8yHu/ag/BUJEJAioC+QCR0WkATAIeB1AVXNV9YhXnwbO64bA7tLtkjHGBL6U/ceZ/f0ObuwTQY+IRpW6bV8SQ2sgw2s605l3hoj0AiJUdf45fecBJ4A9QDrwvKoeAqKBLGCOiKwTkVkiEuL0+S3wnIhkAM8Dj1AMEZnoXGpKyMrK8mE3jDEmMKgq0+YnUTe4Jr8f0anSt+9LYiju/EXPLBSpAbwIPFhMu75AAdAKiAIeFJFoIAjoDbyqqr3wJI+i2sUk4AFVjQAewDmr+FEAqjNVNU5V48LDw33YDWOMCQxfbt7Pd1uzeODyjoTVr13p2/clMWQCEV7TbTj78k4o0A1YLCJpQD8g3ilA34ynjpCnqvuBpUCcs85MVS2qR8zDkygAJgAfOq/fx5NcjDGmWsjJK2D6/CRimtXn1v7tXInBl8SwGogRkSgRCQbGA/FFC1U1W1XDVDVSVSOBFcBoVU3Ac/lomHiE4Ekayaq6F8gQkaJzpOFAkvN6NzDYeT0M2Fa2XTTGmMAxa0kq6YdOMnV0V2rVdOeOgqCSGqhqvohMBhYCNYHZqpooItOABFWNP0/3GcAcYBOeS1JzVHWDs+xe4E0n2aQCtzvz7wJecgrWOcDEC9gvY4wJOLuPnGLGN9u5ulsLLu1Q/gPw+EpUteRWfi4uLk4TEhLcDsMYY8pk8ltrWZS0jy9/N5iIJvUqfHsiskZV486db3c+G2OMH1iRepD5G/YwaUj7SkkK52OJwRhjXJZfUMjU+ERaN6rLPYPbux2OJQZjjHHbW6vSSd57jMdGdqnQAXh8ZYnBGGNcdOhELi98sZVLOzTlqq4t3A4HsMRgjDGuev6LLRw/nc/UShiAx1eWGIwxxiWbdmXz9qp0JvSPJKZ5qNvhnGGJwRhjXKCqTIlPpEm9YO6/vHIG4PGVJQZjjHHB/37YxZqdh/njiM40rFs5A/D4yhKDMcZUsuOn83nqs2R6tGnIuD6VNwCPr0p8JIYxxpjy9Y+vU9h/7DT/vLVPpQ7A4ys7YzDGmEqUmnWc179PZVyfNvRq29jtcIplicEYYyrR9PlJ1AmqyR9HdHY7lJ9kicEYYyrJV5v38c2WLO6/PIbw0MofgMdXlhiMMaYSnM4vYNr8JDo0q8+EAZFuh3NelhiMMaYSzFqyg50HTzJlVKxrA/D4yr+jM8aYKmBP9in+8XUKV8Y257IY/x+j3hKDMcZUsKc+S6ZAlcdGxrodik8sMRhjTAVateMQ8et3c8+gaNcH4PGVT4lBREaIyBYRSRGRh8/TbpyIqIjEOdO1RGSuiGwUkc0i8ohX20YiMk9Ekp1l/b2W3etsL1FEni3LDhpjjFsKCj3PQ2rVsA6ThnRwOxyflXjns4jUBGYAVwCZwGoRiVfVpHPahQL3ASu9Zt8A1FbV7iJSD0gSkbdVNQ14CVigquNEJBio56xnKDAGuEhVT4tIszLvpTHGuOCtVels3nOUGTf3pm6w+wPw+MqXM4a+QIqqpqpqLvAOng/uc00HngVyvOYpECIiQUBdIBc4KiINgEHA6wCqmquqR5w+k4CnVfW0s2x/6XfLGGPcdfhELi98sYX+0U25prt/DMDjK18SQ2sgw2s605l3hoj0AiJUdf45fecBJ4A9QDrwvKoeAqKBLGCOiKwTkVkiEuL06QhcJiIrReRbEbm4uKBEZKKIJIhIQlZWlg+7YYwxleeFRVs4lpPPlNGxfjMAj698SQzF7ZGeWShSA3gReLCYdn2BAqAVEAU8KCLReC5h9QZeVdVeeJJHUe0iCGgM9AN+D7wnxRxVVZ2pqnGqGhce7v8//zLGVB+Ju7N5a2U6t/ZrR+cWDdwOp9R8SQyZQITXdBtgt9d0KNANWCwiaXg+0OOdAvTNeOoIec4loaVAnLPOTFUtqkfMw5Moirb3oXqsAgqBsAvZOWOMqWyqyhPxSTSqF8wDl3d0O5wL4ktiWA3EiEiUUyQeD8QXLVTVbFUNU9VIVY0EVgCjVTUBz+WjYeIRgidpJKvqXiBDRDo5qxkOFBWz/wcMAxCRjkAwcKCsO2qMMZUhfv1uVqUd4vdXdaJhPf8agMdXJf4qSVXzRWQysBCoCcxW1UQRmQYkqGr8ebrPAOYAm/BckpqjqhucZfcCbzrJJhW43Zk/G5gtIpvwFKsnqKpijDF+7sTpfP762Wa6t27IjXERJXfwUz4N1KOqnwGfnTPv8Z9oO8Tr9XE8P1ktrt0PeC4rnTs/F/iFL3EZY4w/mfFNCvuOnuaVW/pQ0w8H4PGV3flsjDHlIO3ACWYt2cHY3q3p084/B+DxlSUGY4wpB9PnJxEcVIOH/XgAHl9ZYjDGmDL6Jnk/XyXv577hHWjWoI7b4ZSZJQZjjCmDogF4osNCuG1AlNvhlAtLDMYYUwZzlqax48AJHh8VS3BQ1fhIrRp7YYwxLth3NIeXv9rG5V2aM6RT1XnepyUGY4y5QE9/nkxeofLYyC5uh1KuLDEYY8wFSEg7xEfrdjHxsmjaNQ0puUMAscRgjDGlVDQAT8uGdfj10PZuh1PuLDEYY0wpvbs6g8TdR/nTNV2oF+zTAyQCiiUGY4wphSMnc3luYTJ9o5ow8qKWbodTISwxGGNMKby4aCvZp/KYOqprwA3A4ytLDMYY46PNe47ynxU7+UW/dsS2CrwBeHxlicEYY3ygqkyNT6Rh3Vr87orAHIDHV5YYjDHGB/M37GHljkM8dFUnGtULdjucCmWJwRhjSnAy1zMAT9dWDRh/cVu3w6lwVe93VsYYU85e+WY7e7JzePmmXgE9AI+v7IzBGGPOY+fBE8z8LpXrerYiLrKJ2+FUCp8Sg4iMEJEtIpIiIg+fp904EVERiXOma4nIXBHZKCKbReQRr7aNRGSeiCQ7y/qfs66HnHWFXejOGWNMWf3l080E1RQeuaZqPQ/pfEpMDCJSE5gBXA3EAjeJSGwx7UKB+4CVXrNvAGqranegD3C3iEQ6y14CFqhqZ6AHsNlrXRHAFUB66XfJGGPKx7dbs1iUtI97h8XQvAoMwOMrX84Y+gIpqpqqqrnAO8CYYtpNB54FcrzmKRAiIkFAXSAXOCoiDYBBwOsAqpqrqke8+r0I/MHpb4wxlS43v5AnPkkkKiyEOwZGuh1OpfIlMbQGMrymM515Z4hILyBCVeef03cecALYg+fb//OqegiIBrKAOSKyTkRmiUiIs67RwC5VXX++oERkoogkiEhCVlaWD7thjDG+e2PZDlKzTvD4yFhqB9V0O5xK5UtiKK4Ef+abvIjUwPMN/8Fi2vUFCoBWQBTwoIhE4/k1VG/gVVXthSd5PCwi9YBHgcdLCkpVZ6pqnKrGhYeH+7Abxhjjm/1Hc3jpy20M69yMoZ2rzgA8vvIlMWQCEV7TbYDdXtOhQDdgsYikAf2AeKcAfTOeOkKequ4HlgJxzjozVbWoHjEPT6JojyeBrHfW1QZYKyItLmz3THlJ2X+MRz7cyITZq8jJK3A7HGMq1NMLkskrUB4f+aNyarXgy30Mq4EYEYkCdgHj8XzgA6Cq2cCZXw6JyGLgIVVNEJHhwDAR+S9QD0/S+Luq7hWRDBHppKpbgOFAkqpuBJp5rSsNiFPVA2XcT3MBVJVl2w8ya0kq32zJIjioBrn5hby6eDsPVPFHApjqa83Ow3y4dheThrQnMqxqDcDjqxITg6rmi8hkYCFQE5itqokiMg1IUNX483SfAcwBNuG5JDVHVTc4y+4F3hSRYCAVuL0M+2HK0en8Aj5Zv4dZS1JJ3nuMsPrBPHB5R37Rry1TP0ni1W+3c33vNrRtWs/tUI0pVwWFnuchNW9Qm8lDO7gdjmtENfB/+BMXF6cJCQluhxHwDp3I5a2VO5m7fCdZx07TqXkod14WxegerahTy1N825udw/AXFtO/fVNmTbjY5YiNKV/vrErn4Q838tL4nozp2brkDgFORNaoaty58+2RGIbtWceZ/f0OPlibSU5eIYM7hvOrG6MY2CHsR8+bb9GwDvdfHsNfP0vmq837GN6luUtRG1O+sk/l8ezCLVwc2ZjRPVq5HY6rLDFUU6rK8tSDvL5kB18l7yc4qAZje7XmjoFRdGweet6+t18axXsJmTzxSRKXdgg7czZhTCB7cdFWjpzMZerovlV2AB5fWWKoZnLzC5m/YTezluwgac9RmoYE89vLY/hFv3aE1a/t0zpq1azBtNFduXnWSv75bSr3Xx5TwVEbU7G27D3Gf1bs5Ka+benaqqHb4bjOEkM1ceRkLm+uTGfusjT2HztNTLP6PHN9d8b0bH1B3/gHdAhj5EUteWVxCmN7tyaiiRWiTWAqGoCnfu0gHrqyk9vh+AVLDFVcatZx5ixNY96aTE7lFXBZTBjP3dCDQTE/rh+U1qPXduHr5P088UkSsyb8qH5lTED4fNNelqceZPqYrjQOqdoD8PjKEkMVpKqs3HGIWUt28FXyPmrVqMF1vVpxx8AoOrcov3FqWzasy33DY3j682S+Tt7HsM5WiDaB5VRuAU9+upkuLRtw8yXt3A7Hb1hiqEJy8wv5bOMeZn2fyqZdR2kSEsy9w2K4tV87wkN9qx+U1h2XRvF+QgZT45MY0N4K0SawvPrtdnYdOcXfbuxRLQbg8ZUlhiog+2Qeb63y1A/2Hs2hfXgIT43tzs96XVj9oDSCg2owbUw3bpm1kpnfpXLfcCtEm8CQcegkr327ndE9WnFJdFO3w/ErlhgCWNqBE8xZuoP3Ejz1g4Edwnjq+u4MjgmnRiV++7m0QxjXdm/JjG9S+FkvK0SbwPCXT5OoKcIj13R2OxS/Y4khwKgqq3Yc4vXvd7Bo8z6CaghjerbmzoFRdGlZfvWD0vrzyC58s2U/0+Yn8a9fWiHa+Lcl27JYmLiP31/ViZYN67odjt+xxBAg8gqc+sGSHWzclU3jerWYPLQDt/ZrRzM/GFmqZcO63DsshmcWJPPNlv0M7VT9HlVsAkNeQSFPfJJEu6b1uHNglNvh+CVLDH4u+2Qeb6/21A/2ZOcQHR7Ckz/rxthebagb7F+F3jsHRvH+mgymxifS/7dNrRBt/NLcZWmk7D/OrF/G2Xv0J1hi8FM7D55gztI03kvI4GRuAQPaN+XJn3VjSMdmlVo/KI3goBo8Mbort76+in99l8q9Vog2fibr2Gle+nIbQzqFM7yLndX+FEsMfkRVSdh5mFlLUvkiyVM/GNWjFXcOjAqY2/Qviwnnmu4tmLE4heusEG38zLMLksnJL+DxkbHV/nlI52OJwQ/kFRTy+aa9vL4klfWZ2TSqV4tfD2nPL/tH0twP6gel9edrY/kmOYvp85OYaYVo4yfWpR/m/TWZ3D04mujw+m6H49csMbgo+1Qe765O542laezOziEqLITp13Xj+t6tqRccuP80rRrV5d7hHXh2wRYWb9nPECtEG5cVOgPwNAutzb3D7BJnSQL30yeAZRw6yeylO3hvdQYncgvoF92EaWO6Mayz/9YPSutXA6OZl5DJ1PhEFj7QlNpBVuQz7pm3JpP1mdm8+PMe1K9tH3slsSNUSVSVtemHmbVkBwsT91JD/q9+0K11YNQPSiM4qAZTR3fll7M9hejJ9i3NuCT7VB7PLEimd9tGXFcNRmUrDz4lBhEZAbyEZ8znWar69E+0Gwe8D1ysqgkiUguYBfR2tvVvVX3KadvIWdYNUOAOVV0uIs8Bo4BcYDtwu6oeKcM+uiq/oJAFiXuZtWQHP2QcoUGdIO4e3J4J/SNp0TDw6gelMahjOFd3a8E/vvEUots0tkK0qXz/76ttHDqZy9w7bAAeX9UoqYGI1ARmAFcDscBNIhJbTLtQ4D5gpdfsG4Daqtod6APcLSKRzrKXgAWq2hnoAWx25i8CuqnqRcBW4JHS75b7jubkMWtJKoOfW8zkt9Zx5GQu08Z0Zfkjw/njiM5VPikU+fPIWAThL/M3l9zYmHK2bd8x5i5LY/zFbavkmXlF8eWMoS+QoqqpACLyDjAGSDqn3XTgWeAhr3kKhIhIEFAXz1nAURFpAAwCbgNQ1VxnGar6hVf/FcC40u2SuzIOneSNZWm8uzqD46fz6RvVhCmjYhnepXm1fHpj60Z1mTysA88t3MK3W7MY3DHc7ZBMNaGqTP0kkXrBNXnoyo5uhxNQfEkMrYEMr+lM4BLvBiLSC4hQ1fki4p0Y5uFJInuAesADqnpIRHoCWcAcEekBrAHuV9UT52z7DuDd4oISkYnARIC2bdv6sBsVa236YV5fsoPPN+2hhgjXXtSSOwdGcVGbRm6H5rpfXRbFvDWeQvSC315mhWhTKRYm7mVpykGeGN2Vpj4OW2s8SryUBBT3NVfPLBSpAbwIPFhMu75AAdAKiAIeFJFoPAmpN/CqqvYCTgAPn7VRkUeBfODN4oJS1ZmqGqeqceHh7nwLzXeeXzT2laWMfWUZ323L4q5B0Xz3h6G8NL6XJQVH7aCaTB3dlR0HTjBryQ63wzHVQE5eAdPnb6Zzi1BuucT9L46Bxpczhkwgwmu6DbDbazoUTwF5sVPYaQHEi8ho4GY8dYQ8YL+ILAXigO+ATFUtqkfMwysxiMgEYCQwXFXPJCF/cSwnj/cSMpmzdAeZh0/Rtkk9po6K5Ya4CELsp3DFGtwxnBFdW/Dy19u4rldrWjeyJ1qaivOaMwDP23f1I6imL99/jTdfjthqIEZEokQkGBgPxBctVNVsVQ1T1UhVjcRTFxitqglAOjBMPEKAfkCyqu4FMkSkaOTt4Tg1C+cXUH901nGyfHazfOw6coonP01iwFNfM31+Ei0b1uG1X/Thm4eGcNulUZYUSvDYKM9vFv4y/9zylDHlJ/PwSV5dvJ1rL2pJ//Y2AM+FKPGTTFXzRWQysBDPz1Vnq2qiiEwDElQ1/jzdZwBzgE14LknNUdUNzrJ7gTedZJMK3O7M/wdQG1jknIGsUNV7Sr9r5eeHjCPMWpLK55v2AnBtd0/9oEeEXSoqjdaN6jJ5aAee/2Ir323NYpAVok0FePLTzYjAn67p4nYoAUv88EpNqcXFxWlCQkK5rrOgUFmU5Ln/IGHnYULrBHFz37b8ckCkXQYpg9P5BVz14nfUEOFzK0SbcrY05QC3zFrJg1d0tKf7+kBE1qjqjx5oZtc+znH8dD7vJ2Qwe+kOMg6dIqJJXaY49QO7lb7sigrRt81Zzevf7+DXQzq4HZKpIjwD8CQS0aQudw2KdjucgGafdI7dR04xd1kab61K51hOPnHtGvPoNV24IrZFtbz/oCIN6dSMK2Ob8/JXKVzXszWt7AzMlIP/LN/J1n3HmXlrHxuAp4yqfWLYkHmEWUt28OnGPQBc3a0Fdw6Molfbxi5HVrU9NjKWK178lr98msQrt/RxOxwT4A4cP82LX27lspgwroht7nY4Aa9aJ4YpH29i7vKdhNYO4o5LI5kwINKe51NJIprU4zdDOvDCoq0s2ZbFZTFWiDYX7rkFWziVW8CUUV3teUjloFonhqGdm9G2aQg3xrUhtE4tt8Opdu4aFM28tZlMiU9kwf2DCA6y35ub0lufcYT31mTwq4FRdGhmA/CUh2r9f+KQTs24c2CUJQWX1KnlKUSnZp3g9e/tjmhTeoWFypT4RJqG1OY++xVSuanWicG4b2inZlwR25yXv97G7iOn3A7HBJgP1+3ih4wjPHx1Z/uCV44sMRjXPT4yloJC5clP7dHcxndHc/J4+vNkerVtxNheNgBPebLEYFwX0aQevxnagU837uH7bQfcDscEiJe/2sbBE6eZOqprlRkS119YYjB+YeKgaNo1rceU+E3k5he6HY7xcyn7jzNnaRo39omwR9NUAEsMxi/UqVWTKaNi2Z51gtlLrRBtfpqq8sQnidQNrsnvR3QquYMpNUsMxm8M69ycy7s05/99tY092VaINsVblLSPJdsO8MDlHQmzAXgqhCUG41emjLJCtPlpOXkFTP80iY7N63Nr/3Zuh1NlWWIwfiWiST0mDWnP/A17WJpihWhztn99l0rGoVNMGdWVWjYAT4WxI2v8zj2D29O2ST2mxCdaIdqcsevIKWYsTuHqbi24tEOY2+FUaZYYjN8pKkSn7D/OG8usEG08/vrZZlTh0WttAJ6KZonB+KXhXZozvHMz/v7lNvZm57gdjnHZsu0H+HTDHiYNaW8PuqwElhiM35oyqiv5hcqTn1khujrLLyjkifgkWjeqyz2D27sdTrXgU2IQkREiskVEUkTk4fO0GyciKiJxznQtEZkrIhtFZLOIPOLVtpGIzBORZGdZf2d+ExFZJCLbnL82MEI11bZpPSYNbs8n63ezbLsVoqurN1ems2XfMR4b2cUG4KkkJSYGEakJzACuBmKBm0Qktph2ocB9wEqv2TcAtVW1O9AHuFtEIp1lLwELVLUz0AMo+lr4MPCVqsYAXznTppqaNKS9Z3jVjxPJK7BCdHVz8PhpXvhiCwM7hHFV1xZuh1Nt+HLG0BdIUdVUVc0F3gHGFNNuOvAs4H1BWIEQEQkC6gK5wFERaQAMAl4HUNVcVT3i9BkDzHVezwWuK90umaqkTq2aTBnZlW37j/PG0jS3wzGV7PkvtnIyt4Apo2JtAJ5K5EtiaA1keE1nOvPOEJFeQISqzj+n7zzgBLAHSAeeV9VDQDSQBcwRkXUiMktEQpw+zVV1D4Dzt1lxQYnIRBFJEJGErKwsH3bDBKrLY5szrHMz/v7lVvYdtUJ0dbExM5t3VqczYUAkMc1D3Q6nWvElMRSXpvXMQpEawIvAg8W06wsUAK2AKOBBEYnGM3Jcb+BVVe2FJ3mU6pKRqs5U1ThVjQsPt2Ehq7opo2LJszuiqw1VZUr8JpqGBHP/5TYAT2XzJTFkAhFe022A3V7ToUA3YLGIpAH9gHinAH0znjpCnqruB5YCcc46M1W1qB4xD0+iANgnIi0BnL/7L2THTNXSrmkI9wxuT/z63SzfftDtcEwF+2jdLtamH+EPV3WmgQ3AU+l8SQyrgRgRiRKRYGA8EF+0UFWzVTVMVSNVNRJYAYxW1QQ8l4+GiUcInqSRrKp7gQwRKXo04nAgyXkdD0xwXk8APi7bLpqq4tdD2tOmcV0e/3iTFaKrsOOn83nq82R6tGnIuD5t3A6nWioxMahqPjAZWIjnl0PvqWqiiEwTkdEldJ8B1Ac24Ukwc1R1g7PsXuBNEdkA9AT+6sx/GrhCRLYBVzjTxjh3RHsK0XOXpbkdjqkgL3+1jaxjp5k62gbgcYuoasmt/FxcXJwmJCS4HYapBKrKHW+sZnXaYb56cDDNG9RxOyRTjrZnHWfE37/jup6tee6GHm6HU+WJyBpVjTt3vt35bAKKiDBlVFdy8wv5q90RXaWoKtM+SaJOUE3+MKKz2+FUa5YYTMCJDAvhnsHRfPzDblakWiG6qpi1ZAffbs3i/stjCA+1AXjcZInBBKRJQzrQupHdEV0VFBYqf5mfxJOfbWZE1xZMGBDpdkjVniUGE5DqBtfk8VGxbNl3zArRASw3v5DfvvsDs77fwS/7t2PGLb1tAB4/YP8CJmBdGducIZ3C+fuX29hvd0QHnGM5edzxxmri1+/m91d14onRXalpv0LyC5YYTMASEaY6heinPk92OxxTCvuP5fDzf65geepBnht3Eb8Z2sGeheRHLDGYgBYZFsLEQdF8tG4XK60QHRBSs45z/avL2HHgBLMmxHFDXETJnUylssRgAt5vhjqF6PhE8q0Q7dfWpR9m3GvLOXG6gLcn9mNop2KfkWlcZonBBLy6wTV5bGQsyXuP8e/lO90Ox/yEr5P3cfO/VhJSuyYfTBpAz4hGbodkfoIlBlMlXNW1OYM6hvPioq3sP2aFaH/zXkIGd/17DdHhIXwwaQBRYSEldzKuscRgqgQR4YnRXTmdX8jTn1kh2l+oKi9/tY0/zNvAgPZNeffu/jQLtceY+DtLDKbKiAoL4a5BUXy4bherdhxyO5xqr6BQeezjTbywaCvX9WzF6xMupn7tILfDMj6wxGCqlKJC9OMfb7JCtIty8gr4zcIR6M8AABQJSURBVJtr+e+KdO4eFM3fbuxJcJB93AQK+5cyVUq94CAeG9mF5L3H+M8KK0S7IftkHre+vpIFiXt5bGQsj1zTxR6fHWAsMZgq56quLbgsJoy/fWGF6Mq2+8gpxr22jPUZ2bx8Uy/uHBjldkjmAlhiMFVOUSE6J7+Ap+2O6EqzZe8xxr6yjD3ZObxx+8WM6tHK7ZDMBbLEYKqk6PD63HVZNB+u3cXqNCtEV7SVqQe54bVlFKry3t39GdAhzO2QTBlYYjBV1uRhHWjVsA6P/c8K0RXp8417uHX2KsJCa/PBpAHEtmrgdkimjHxKDCIyQkS2iEiKiDx8nnbjRERFJM6ZriUic0Vko4hsFpFHvNqmOfN/EJEEr/k9RWRF0XwR6VuWHTTVl6cQ7bkj+r9WiK4Q/16exq/fWkvXVg344J4BRDSp53ZIphyUmBhEpCYwA7gaiAVuEpHYYtqFAvcBK71m3wDUVtXuQB/gbhGJ9Fo+VFV7njPm6LPAE6raE3jcmTbmgozo5ilEv7BoK1nHTrsdTpWhqjy3MJnHP05kWKdmvPWrfjQOCXY7LFNOfDlj6AukqGqqquYC7wBjimk3Hc+HuPfPQBQIEZEgoC6QCxwtYXsKFJ2LNgR2+xCjMcUSEaaO7kpOnhWiy0teQSG/n7eBGd9sZ/zFEfzz1j7UDa7pdlimHPmSGFoDGV7Tmc68M0SkFxChqvPP6TsPOAHsAdKB51W1qBKowBciskZEJnr1+S3wnIhkAM8Dj1AMEZnoXGpKyMrK8mE3THXVPrw+v7osmg/WZrJmpxWiy+Jkbj53/TuBeWsyuX94DE+N7U6QjbhW5fjyL1rcnSl6ZqFIDeBF4MFi2vUFCoBWQBTwoIhEO8suVdXeeC5R/UZEBjnzJwEPqGoE8ADwenFBqepMVY1T1bjw8HAfdsNUZ/cO60DLhnV47H/2aO4LdfD4aW6auYLvtmbx5M+68cAVHW1wnSrKl8SQCXiPpNGGsy/vhALdgMUikgb0A+KdAvTNwAJVzVPV/cBSIA5AVXc7f/cDH+FJIgATgA+d1+97zTfmgtULDuLP18aStOcob65MdzucgJN+8CTXv7qM5L3HeO0XfbjlknZuh2QqkC+JYTUQIyJRIhIMjAfiixaqaraqhqlqpKpGAiuA0aqagOfy0TDxCMGTNJJFJMQpVuPMvxLY5KxyNzDYeT0M2FbmvTQGuKZ7CwZ2COP5L7Zw4LgVon21aVc2Y19dxuGTebx11yVc2bWF2yGZClZiYlDVfGAysBDYDLynqokiMk1ERpfQfQZQH8+H/mpgjqpuAJoD34vIemAV8KmqLnD63AW84Cz7KzDxx6s1pvS8C9HPWCHaJ0u2ZfHzfy6ndlANPpjUnz7tmrgdkqkEoqolt/JzcXFxmpCQUHJDY4CnPt/MP79N5YNJA+jTrrHb4fit/63bxUPvr6dDs/q8cXtfWjS0cRSqGhFZc87tAoDd+WyqofuGxdCiQR0e/3gTBYWB/8WovKkqM7/bzm/f/YE+7Rrz7t39LSlUM5YYTLUTUjuIP4/sQuLuo7y10u6I9lZYqEyfv5m/fpbMtd1bMveOvjSsW8vtsEwls8RgqqVru7dkQPumPLdwCwetEA3A6fwC7ntnHbOX7uC2AZG8fFMv6tSyG9eqI0sMploSEaaN6crJ3AKeWWCF6KM5edw2ezXzN+zh4as7M2VUrA2uU41ZYjDVVodmodw5MIr3EjJZm37Y7XBcs+9oDje+tpzVaYf42409uGdwe7txrZqzxGCqtXuHx9C8Qe1qW4jennWcsa8sI/3QSV6/7WLG9m7jdkjGD1hiMNVa/dqeO6I37TrKW6uq1x3Ra3Ye5vpXl5GTV8A7E/sxuKM9WsZ4WGIw1d7Ii5xC9ILkalOI/jJpH7fMWkHDurX4YNIALmrTyO2QjB+xxGCqvaIxok/mFvDsgi1uh1Ph3lmVzsT/JNCxeSgfTBpAZFiI2yEZP2OJwRggpnkodwyM4t2EjCpbiFZVXvpyGw9/uJGBMeG8fVc/wurXdjss44csMRjjuK8KF6LzCwr500ebePHLrYzt3ZrXJ8QRUjvI7bCMn7LEYIyjfu0gHnUK0W9XoUJ0Tl4Bk95cy9ur0pk0pD0v3NCDWja4jjkPe3cY42XURS3pH+25I/rQiVy3wymzIydzuWXWSr7cvI+po2L544jOdo+CKZElBmO8iAhPjOnKidP5PBvgd0RnHvYMrrMxM5t/3NSb2y6NcjskEyAsMRhzjo7NQ7n90kjeTcjgh4wjbodzQZL3HuX6V5ex/9hp5t7Rl2svaul2SCaAWGIwphj3X96R8PqBWYhevv0gN7y6HID37+lP//ZNXY7IBBpLDMYUw1OI7sKGzGzeWR04hehPN+xhwuxVNG9Yhw9/fSmdWzRwOyQTgCwxGPMTRvdoxSVRTXhu4RYOB0Ah+o2lO5j89lq6t2nIvHv607pRXbdDMgHKp8QgIiNEZIuIpIjIw+dpN05EVETinOlaIjJXRDaKyGYRecSrbZoz/wcRSThnPfc620sUkWcvdOeMKQvPo7m7cSwnn2cX+u8d0arK058nM/WTJC7v0pw3f3UJjeoFux2WCWAl3uEiIjWBGcAVQCawWkTiVTXpnHahwH3ASq/ZNwC1VbW7iNQDkkTkbVVNc5YPVdUD56xnKDAGuEhVT4tIswvcN2PKrFOLUG4bEMnspTsYf3EEPSL865lCeQWF/PGDDXy4dhc3X9KWaaO7EmT3KJgy8uUd1BdIUdVUVc0F3sHzwX2u6cCzQI7XPAVCRCQIqAvkAkdL2N4k4GlVPQ2gqvt9iNGYCvPby2MIcwrRhX5UiD5xOp875ybw4dpd/O6Kjjx5XTdLCqZc+PIuag1keE1nOvPOEJFeQISqzj+n7zzgBLAHSAeeV9VDzjIFvhCRNSIy0atPR+AyEVkpIt+KyMXFBSUiE0UkQUQSsrKyfNgNYy5MaJ1aPHpNF9ZnZvNuQkbJHSpB1rHTjJ+5gu+3ZfH02O7cNzzGblwz5caXxFDcu+3M1yYRqQG8CDxYTLu+QAHQCogCHhSRaGfZparaG7ga+I2IDHLmBwGNgX7A74H3pJh3vKrOVNU4VY0LD7fnyJuKNaZnK/pGNeGZBcmuF6J3HjzBuNeWsW3/MWbeGsf4vm1djcdUPb4khkwgwmu6DbDbazoU6AYsFpE0PB/o8U4B+mZggarmOZeElgJxAKq62/m7H/gITxIp2t6H6rEKKATCLmz3jCkfRWNEH8vJ57kv3CtEb8g8wthXlnH0VB5v3dWPy2ObuxaLqbp8SQyrgRgRiRKRYGA8EF+0UFWzVTVMVSNVNRJYAYxW1QQ8l4+GiUcInqSRLCIhTrEaZ/6VwCZnlf8DhjnLOgLBwFkFamPc0LlFAyb0j+TtVelsyKz8O6K/3ZrF+JkrqFOrJvMmDaB328aVHoOpHkpMDKqaD0wGFgKbgfdUNVFEponI6BK6zwDq4/nQXw3MUdUNQHPgexFZD6wCPlXVBU6f2UC0iGzCU+ieoKr+U/Ez1dpvr4ihaUhtHvs4sVIL0R+uzeTON1bTrmkIH/56AO3D61fatk31I1XhMzcuLk4TEhJKbmhMOfhoXSYPvLuep8d2r/Dr+6rKa9+m8syCZAa0b8prt/ahQZ1aFbpNU32IyBpVjTt3vv22zZhSuq5na/pGegrRR05WXCG6oFB54pMknlmQzKgerZhz+8WWFEylsMRgTCkVPZr7aE4+z1XQHdE5eQXc9/Y63liWxp0Do3jp5z2pHVSzQrZlzLksMRhzAbq0bMAv+7fjrVXpbMzMLtd1Z5/KY8LsVXy6cQ+PXtOFx0bGUqOG3aNgKo8lBmMu0ANXdHQK0eV3R/Te7Bx+/s/lrE0/zN9/3pO7BkWX3MmYcmaJwZgL1KBOLf50TWd+yDjC+2vKfkd0yv5jjH1lKRmHTjL7tou5rlfrkjsZUwEsMRhTBj/r1ZqLIxvzzIItZSpEr9l5iOtfXU5ugfLu3f25LMbu5jfuscRgTBmICE+M7saRk7k8f4F3RH+RuJeb/7WSJiHBfDhpAN1aNyznKI0pHUsMxpRRbKsG/LJ/JG+uTGfTrtIVot9cuZN7/ruGzi1CmXdPf9o2rVdBURrjO0sMxpQDTyE62OdCtKryt0VbefSjTQzuGM7bE/vRtH7tSojUmJJZYjCmHDSsW4uHr+7CuvQjzFuTed62+QWFPPLhRv7fV9sY16cNM38ZR73gEsfMMqbSWGIwppyM7dWauHaNeXpBMtkn84ptcyq3gLv/s4Z3VmcweWgHnht3EbVscB3jZ+wdaUw5qVHDM0b0kZO5vLDox4XoQydyuXnWCr7esp/pY7ry0FWdbHAd45csMRhTjmJbNeDWfu3474qdZxWiMw6dZNxry0jcfZRXb+nNrf0j3QvSmBJYYjCmnP3uyk40rhd8ZozopN1HGfvqMg4cO81/77yEEd1auh2iMedlicGYcuYpRHdmbfoRpn6SyM//uZygGsK8SQPoG9XE7fCMKZElBmMqwPW929C7bSP+vXwnLRrW4YNJA+jYPNTtsIzxif1GzpgKUKOG8MKNPXlr5U4mD42hYT0bR8EEDksMxlSQqLAQHr021u0wjCk1ny4licgIEdkiIiki8vB52o0TERWROGe6lojMFZGNIrJZRB7xapvmzP9BRH40LqeIPOSsK+xCdswYY8yFKfGMQURqAjOAK4BMYLWIxKtq0jntQoH7gJVes28AaqtqdxGpBySJyNuqmuYsH6qqB4rZZoSzvfQL2CdjjDFl4MsZQ18gRVVTVTUXeAcYU0y76cCzQI7XPAVCRCQIqAvkAkd92OaLwB+c/sYYYyqRL4mhNeA9CkmmM+8MEekFRKjq/HP6zgNOAHvwfPt/XlUPOcsU+EJE1ojIRK91jQZ2qer68wUlIhNFJEFEErKysnzYDWOMMb7wpfhc3D37Z77Ji0gNPN/wbyumXV+gAGgFNAaWiMiXqpoKXKqqu0WkGbBIRJKBBOBR4MqSglLVmcBMgLi4ODuzMMaYcuLLGUMmEOE13QbY7TUdCnQDFotIGtAPiHcK0DcDC1Q1T1X3A0uBOABV3e383Q98hCeJtAeigPXOutoAa0WkxYXuoDHGmNLxJTGsBmJEJEpEgoHxQHzRQlXNVtUwVY1U1UhgBTBaVRPwXD4aJh4heJJGsoiEOMVqnPlXAptUdaOqNvNaVybQW1X3lt8uG2OMOZ8SE4Oq5gOTgYXAZuA9VU0UkWlOPeB8ZgD1gU14EswcVd0ANAe+F5H1wCrgU1VdUIb9MMYYU05ENfAvz4tIFrDzAruHAT/6yawfsLhKx+IqHYurdPw1LihbbO1UNfzcmVUiMZSFiCSoapzbcZzL4iodi6t0LK7S8de4oGJis4foGWOMOYslBmOMMWexxODcC+GHLK7SsbhKx+IqHX+NCyogtmpfYzDGGHM2O2MwxhhzFksMxhhjzlKlE0NJ40iIyG0ikuWMCfGDiPzKa9kEEdnm/DfBj+Iq8Joff27fiozLaXOjiCSJSKKIvOU137XjVUJcrh0vEXnRa9tbReSI1zI331/ni8vN49VWRL4RkXUiskFErvFa9ojTb4uIXOUPcYlIpIic8jper1VyXO1E5CsnpsUi0sZrWdneX6paJf8DagLbgWggGFgPxJ7T5jbgH8X0bQKkOn8bO68bux2Xs+y4i8crBlhXdCyAZn5yvIqNy+3jdU77e4HZ/nC8fiout48XniLqJOd1LJDm9Xo9UBvPs9S2AzX9IK5IPI/ycet4vQ9McF4PA/5TXu+vqnzG4Os4EsW5ClikqodU9TCwCBjhB3FVJF/iuguY4RwT1PMARHD/eP1UXBWptP+ONwFvO6/dPl4/FVdF8iUuBRo4rxvyfw/rHAO8o6qnVXUHkOKsz+24KpIvccUCXzmvv/FaXub3V1VODCWOI+G43jkVmyeekeNK07ey4wKoI55xKFaIyHXlFJOvcXUEOorIUmf7I0rR1424wN3jBXhO+fF80/26tH0rOS5w93hNBX4hIpnAZ3jOZnzt60ZcAFHOJaZvReSycorJ17jWA9c7r38GhIpIUx/7nldVTgznHUfC8QkQqaoXAV8Cc0vR1424ANqq5/b3m4G/i0j7SowrCM9lmyF4vmnOEpFGPvZ1Iy5w93gVGQ/MU9WCC+hbWmWJC9w9XjcBb6hqG+Aa4D/iGe/F7eP1U3HtwXO8egG/A94SkQaUD1/ieggYLCLrgMHALiDfx77nVZUTQ0njSKCqB1X1tDP5L6CPr31digv9v3EsUoHFQK/Kistp87F6xtfYAWzB84Hs6vE6T1xuH68i4zn7co3bx+un4nL7eN0JvOdsfzlQB88D4tw+XsXG5VzaOujMX4OnJtCxsuJS1d2qOtZJTI8687J93Kfzq4jCiT/8h+dbZCqeU+Wi4k3Xc9q09Hr9M2CF/l/xZgeewk1j53UTP4irMVDbeR0GbOM8hcUKiGsEMNdr+xlAUz84Xj8Vl6vHy2nXCUjDuZnUH95f54nL7ffX58BtzusueD7MBOjK2cXnVMqv+FyWuMKL4sBTJN5Vye/7MKCG8/pJYFp5vb/KvAP+/B+e076teDL5o868aXgGEgJ4Ckh0Dvo3QGevvnfgKXKlALf7Q1zAAGCjM38jcGclxyXA34AkZ/vj/eR4FRuX28fLmZ4KPF1MX9eO10/F5fbxwlNMXeps/wfgSq++jzr9tgBX+0NceK7vF/1/uhYYVclxjcOTvLcCs3CSenm8v+yRGMYYY85SlWsMxhhjLoAlBmOMMWexxGCMMeYslhiMMcacxRKDMcaYs1hiMMYYcxZLDMYYY87y/wGcmgnVLzAcwAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### colsample_bytree=0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)感觉不用调了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 减小学习率，调整n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 3650\n",
      "best cv score: 0.4769280691435139\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':50,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用所有训练数据，采用最佳参数重新训练模型\n",
    "由于样本数目增多，模型复杂度稍微扩大一点？\n",
    "num_leaves增多5\n",
    "min_child_samples按样本比例增加到60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "               colsample_bytree=0.4, importance_type='split',\n",
       "               learning_rate=0.01, max_bin=127, max_depth=7,\n",
       "               min_child_samples=60, min_child_weight=0.001, min_split_gain=0.0,\n",
       "               n_estimators=3650, n_jobs=4, num_class=9, num_leaves=75,\n",
       "               objective='multiclass', random_state=None, reg_alpha=0.0,\n",
       "               reg_lambda=0.0, silent=False, subsample=0.8,\n",
       "               subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          'n_estimators':n_estimators_2,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':75,\n",
    "          'min_child_samples':60,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "lg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickle.dump(lg, open(\"Otto_LightGBM_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\"columns\":list(feat_names), \"importance\":list(lg.feature_importances_.T)})\n",
    "df = df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>feat_67_tfidf</td>\n",
       "      <td>51271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>feat_25_tfidf</td>\n",
       "      <td>50311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>feat_24_tfidf</td>\n",
       "      <td>48271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>feat_48_tfidf</td>\n",
       "      <td>46077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>feat_40_tfidf</td>\n",
       "      <td>43884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>feat_61_tfidf</td>\n",
       "      <td>3117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>feat_82_tfidf</td>\n",
       "      <td>3040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>feat_84_tfidf</td>\n",
       "      <td>2546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>feat_6_tfidf</td>\n",
       "      <td>1804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>feat_51_tfidf</td>\n",
       "      <td>1452</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>93 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          columns  importance\n",
       "66  feat_67_tfidf       51271\n",
       "24  feat_25_tfidf       50311\n",
       "23  feat_24_tfidf       48271\n",
       "47  feat_48_tfidf       46077\n",
       "39  feat_40_tfidf       43884\n",
       "..            ...         ...\n",
       "60  feat_61_tfidf        3117\n",
       "81  feat_82_tfidf        3040\n",
       "83  feat_84_tfidf        2546\n",
       "5    feat_6_tfidf        1804\n",
       "50  feat_51_tfidf        1452\n",
       "\n",
       "[93 rows x 2 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAASA0lEQVR4nO3df6zddX3H8efLVhR1WJBCuhZ2MTZOJBGwgW4siwODBYzlD8ggThqCaWIgw8VFi/8Qf5BgsogjUZJGOovZRIIaGkG7pkC2JYoUcWCppHfI4K4drSsgmxGGvvfH+Vx3uD3tvbe99557z3k+kpv7/b6/n3Pu55x+732dz+f7o6kqJEnD7XX97oAkqf8MA0mSYSBJMgwkSRgGkiRgcb87cKROPPHEGhkZ6Xc3JGnBeOSRR35RVUt7bVuwYTAyMsKOHTv63Q1JWjCS/PuhtjlNJEkyDCRJhoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkFvAVyJLmp5EN9/5u+embL+ljTzQdjgwkSVMLgyRPJ3k8yU+S7Gi1E5JsS7K7fT++1ZPk1iSjSR5LcnbX86xr7XcnWddVf297/tH22Mz0C5UkHdp0RgZ/VlVnVtWqtr4B2F5VK4HtbR3gImBl+1oP3Aad8ABuBM4FzgFuHA+Q1mZ91+PWHPErkiRN29FME60FNrflzcClXfU7quOHwJIky4APANuq6kBVPQ9sA9a0bcdV1Q+qqoA7up5L88zIhntfMycsaTBMNQwK+MckjyRZ32onV9VegPb9pFZfDjzb9dixVjtcfaxH/SBJ1ifZkWTH/v37p9h1SdJkpno20XlVtSfJScC2JD87TNte8/11BPWDi1UbgY0Aq1at6tlGkjR9UxoZVNWe9n0f8B06c/7PtSke2vd9rfkYcErXw1cAeyapr+hRlyTNkUnDIMmbk/ze+DJwIfBTYAswfkbQOuCetrwFuKqdVbQaeLFNI20FLkxyfDtwfCGwtW17KcnqdhbRVV3PJUmaA1OZJjoZ+E4723Mx8A9V9f0kDwN3JbkGeAa4vLW/D7gYGAV+BVwNUFUHknwOeLi1+2xVHWjLHwO+BhwLfK99SZLmyKRhUFVPAe/pUf8v4IIe9QKuPcRzbQI29ajvAM6YQn8lSbPAK5AlSd6bSIfnfWak4eDIQJJkGEiSDANJEoaBJAnDQJKEYSBJwjCQJGEYSJLwojNp1njBnhYSRwaSJMNAkmQYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGGgITSy4d7XXBAmyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSUwjDJIsSvJoku+29dOSPJRkd5JvJjmm1d/Q1kfb9pGu57ih1Z9M8oGu+ppWG02yYeZeniTNnYV836vpjAyuB3Z1rX8BuKWqVgLPA9e0+jXA81X1DuCW1o4kpwNXAO8G1gBfaQGzCPgycBFwOnBlaytJmiNTCoMkK4BLgK+29QDnA3e3JpuBS9vy2rZO235Ba78WuLOqXq6qnwOjwDnta7SqnqqqV4A7W1tJ0hyZ6sjgS8Angd+29bcBL1TVq219DFjelpcDzwK07S+29r+rT3jMoeoHSbI+yY4kO/bv3z/FrkuSJjNpGCT5ILCvqh7pLvdoWpNsm2794GLVxqpaVVWrli5depheS5KmY/EU2pwHfCjJxcAbgePojBSWJFncPv2vAPa09mPAKcBYksXAW4EDXfVx3Y85VF2SNAcmHRlU1Q1VtaKqRugcAL6/qj4MPABc1pqtA+5py1vaOm37/VVVrX5FO9voNGAl8CPgYWBlOzvpmPYztszIq1vgxs9MWKhnJ0haOKYyMjiUTwF3Jvk88Chwe6vfDnw9ySidEcEVAFW1M8ldwBPAq8C1VfUbgCTXAVuBRcCmqtp5FP2SJE3TtMKgqh4EHmzLT9E5E2him18Dlx/i8TcBN/Wo3wfcN52+SJJmjlcgS5IMA0mSYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpImGNlwLyMb7u13NzTHDANJkmEgSTIMJEkYBpIkDANJEoaBJAnDQJKEYSBJYgphkOSNSX6U5F+T7EzymVY/LclDSXYn+WaSY1r9DW19tG0f6XquG1r9ySQf6KqvabXRJBtm/mVKkg5nKiODl4Hzq+o9wJnAmiSrgS8At1TVSuB54JrW/hrg+ap6B3BLa0eS04ErgHcDa4CvJFmUZBHwZeAi4HTgytZWkjRHJg2D6vjvtvr69lXA+cDdrb4ZuLQtr23rtO0XJEmr31lVL1fVz4FR4Jz2NVpVT1XVK8Cdra0kaY5M6ZhB+wT/E2AfsA34N+CFqnq1NRkDlrfl5cCzAG37i8DbuusTHnOouibwnjGSZsuUwqCqflNVZwIr6HySf1evZu17DrFtuvWDJFmfZEeSHfv375+845KkKVk8ncZV9UKSB4HVwJIki9un/xXAntZsDDgFGEuyGHgrcKCrPq77MYeqT/z5G4GNAKtWreoZGMOie4Tw9M2X9LEnkgbBVM4mWppkSVs+Fng/sAt4ALisNVsH3NOWt7R12vb7q6pa/Yp2ttFpwErgR8DDwMp2dtIxdA4yb5mJFydJmpqpjAyWAZvbWT+vA+6qqu8meQK4M8nngUeB21v724GvJxmlMyK4AqCqdia5C3gCeBW4tqp+A5DkOmArsAjYVFU7Z+wVSpImNWkYVNVjwFk96k/ROX4wsf5r4PJDPNdNwE096vcB902hv5KkWeAVyJIkw0CSZBhIkjAMNE95gZ00t6Z1nYGk4eW1LYPNkYEkyTCQJDlNpAXA6Qlp9jkykCQZBpIkp4kkadYthKlORwaSJEcG0riF8OlNmi2ODCRJhoG0UHiLDs0mw0CSZBhIkgwDSRKGgSQJw0CShGEgaR7yzKm5ZxhIkgwDSZK3o5CGkrfe0ESODCRJhoEkyWmiGTc+/HboPT1OW0j95chAkmQYSJIMA0kShoEkCcNAksQUwiDJKUkeSLIryc4k17f6CUm2Jdndvh/f6klya5LRJI8lObvruda19ruTrOuqvzfJ4+0xtybJbLxYSVJvUxkZvAp8oqreBawGrk1yOrAB2F5VK4HtbR3gImBl+1oP3Aad8ABuBM4FzgFuHA+Q1mZ91+PWHP1LkzSfjd+MzhvSzQ+ThkFV7a2qH7fll4BdwHJgLbC5NdsMXNqW1wJ3VMcPgSVJlgEfALZV1YGqeh7YBqxp246rqh9UVQF3dD2XJGkOTOuisyQjwFnAQ8DJVbUXOoGR5KTWbDnwbNfDxlrtcPWxHvVeP389nREEp5566nS6LmnAeKHizJryAeQkbwG+BXy8qn55uKY9anUE9YOLVRuralVVrVq6dOlkXdYQcJpBmhlTCoMkr6cTBH9fVd9u5efaFA/t+75WHwNO6Xr4CmDPJPUVPeqSpDkylbOJAtwO7KqqL3Zt2gKMnxG0Drinq35VO6toNfBim07aClyY5Ph24PhCYGvb9lKS1e1nXdX1XJKkOTCVYwbnAR8BHk/yk1b7NHAzcFeSa4BngMvbtvuAi4FR4FfA1QBVdSDJ54CHW7vPVtWBtvwx4GvAscD32pckaY5MGgZV9S/0ntcHuKBH+wKuPcRzbQI29ajvAM6YrC8LjQe4JC0UXoEsHYIHpzVMDANJkv+5jSTNR3M9zezIQJJkGPSL89GS5hPDQJLkMQNJ/bdQT8NeqP3uxZGBAKetpGFnGOiIGSDS4DAMBpB/pCVNl2EgSTIMJEmeTSRpBjgtufAZBtI8NSinLY6/joX8GoaB00SSJMNAkuQ00UAYlOkESf3jyECSZBhIGgxebHl0DANJkmEgSbNhoY1UDANJkmcTaTB5hpU0PYaBhtpCGsZLs8kw0MA41G0PZmKUMAi3VDD4dDhDecxgoR3YkaTZ5shgiDiPPvj8N9aRGsqRgSTptRwZaEHxk680OxwZSJIcGcw3HtiW1A+TjgySbEqyL8lPu2onJNmWZHf7fnyrJ8mtSUaTPJbk7K7HrGvtdydZ11V/b5LH22NuTZKZfpGSFq7xs//8oDS7pjJN9DVgzYTaBmB7Va0Etrd1gIuAle1rPXAbdMIDuBE4FzgHuHE8QFqb9V2Pm/izJE3gH0jNtEmniarqn5KMTCivBd7XljcDDwKfavU7qqqAHyZZkmRZa7utqg4AJNkGrEnyIHBcVf2g1e8ALgW+dzQvSprvBuEiNh2Z+XoSxJEeMzi5qvYCVNXeJCe1+nLg2a52Y612uPpYj3pPSdbTGUVw6qmnHmHXpdnjH/nZ1a+R0DD8u8702US95vvrCOo9VdXGqlpVVauWLl16hF2UJE10pCOD55Isa6OCZcC+Vh8DTulqtwLY0+rvm1B/sNVX9Gg/LwzDpwEtfB43OHLzdcqmH450ZLAFGD8jaB1wT1f9qnZW0WrgxTadtBW4MMnx7cDxhcDWtu2lJKvbWURXdT2XJGmOTDoySPINOp/qT0wyRuesoJuBu5JcAzwDXN6a3wdcDIwCvwKuBqiqA0k+Bzzc2n12/GAy8DE6ZywdS+fAsQePJWmOTeVsoisPsemCHm0LuPYQz7MJ2NSjvgM4Y7J+aOFyuk3j3BcONl/eE69AlqR5op/HfwyDLh5MkjSsDAPNKQNX/TZfpmXmG8NAGmD+4fMDyFR5C2tJkiODuTLon04G/fVJg84wkI7CMIegU1CDxTCQ5sAwh4YWBo8ZSJIMA0mSYSBJwmMG80I/DsQ5hy2pm2EgzSP+3wTqF8NAM27QRx3+wZ7//DeaPo8ZDLiRDff6iyFpUo4MjtIgfgo2PDTOfWF4GAZTNIh/9CVpnNNEkiRHBg6DJS0Es30K+tCHgSSBU8GGgfrGUZk0f3jMQJJkGEjyehQ5TSRJBxnG4weODCRJjgwkaToGdTrNkYEkyTCQJDlNdEQGdZgoaXg5MpD6zNM6NR8YBpIkw0CSNI/CIMmaJE8mGU2yod/9kaRhMi/CIMki4MvARcDpwJVJTu9vryRpeMyLMADOAUar6qmqegW4E1jb5z5J0tBIVfW7DyS5DFhTVR9t6x8Bzq2q6ya0Ww+sb6vvBJ48ih97IvCLo3j8oPB98D0Y5/sw+O/BH1TV0l4b5st1BulROyilqmojsHFGfmCyo6pWzcRzLWS+D74H43wfhvs9mC/TRGPAKV3rK4A9feqLJA2d+RIGDwMrk5yW5BjgCmBLn/skSUNjXkwTVdWrSa4DtgKLgE1VtXOWf+yMTDcNAN8H34Nxvg9D/B7MiwPIkqT+mi/TRJKkPjIMJEnDGQbDeOuLJKckeSDJriQ7k1zf6ick2ZZkd/t+fL/7OtuSLEryaJLvtvXTkjzU3oNvtpMYBlqSJUnuTvKztk/80ZDuC3/Vfh9+muQbSd44jPsDDGEYDPGtL14FPlFV7wJWA9e2170B2F5VK4HtbX3QXQ/s6lr/AnBLew+eB67pS6/m1t8C36+qPwTeQ+f9GKp9Icly4C+BVVV1Bp2TV65gOPeH4QsDhvTWF1W1t6p+3JZfovPLv5zOa9/cmm0GLu1PD+dGkhXAJcBX23qA84G7W5NheA+OA/4UuB2gql6pqhcYsn2hWQwcm2Qx8CZgL0O2P4wbxjBYDjzbtT7WakMjyQhwFvAQcHJV7YVOYAAn9a9nc+JLwCeB37b1twEvVNWrbX0Y9oe3A/uBv2vTZV9N8maGbF+oqv8A/gZ4hk4IvAg8wvDtD8BwhsGUbn0xqJK8BfgW8PGq+mW/+zOXknwQ2FdVj3SXezQd9P1hMXA2cFtVnQX8DwM+JdRLOyayFjgN+H3gzXSmjyca9P0BGM4wGNpbXyR5PZ0g+Puq+nYrP5dkWdu+DNjXr/7NgfOADyV5ms704Pl0RgpL2jQBDMf+MAaMVdVDbf1uOuEwTPsCwPuBn1fV/qr6X+DbwB8zfPsDMJxhMJS3vmhz47cDu6rqi12btgDr2vI64J657ttcqaobqmpFVY3Q+Xe/v6o+DDwAXNaaDfR7AFBV/wk8m+SdrXQB8ARDtC80zwCrk7yp/X6Mvw9DtT+MG8orkJNcTOcT4fitL27qc5dmXZI/Af4ZeJz/ny//NJ3jBncBp9L55bi8qg70pZNzKMn7gL+uqg8meTudkcIJwKPAX1TVy/3s32xLciadg+jHAE8BV9P5cDhU+0KSzwB/Tudsu0eBj9I5RjBU+wMMaRhIkl5rGKeJJEkTGAaSJMNAkmQYSJIwDCRJGAaSJAwDSRLwf6k7zm4hUGuFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(range(len(lg.feature_importances_)), lg.feature_importances_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finish\n"
     ]
    }
   ],
   "source": [
    "test = pd.read_csv(\"./Otto_FE_test_tfidf.csv\")\n",
    "test_id = test['id']   \n",
    "X_test = test.drop([\"id\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_test.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_test = csr_matrix(X_test)\n",
    "\n",
    "print('finish')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finish\n"
     ]
    }
   ],
   "source": [
    "#load训练好的模型\n",
    "import pickle\n",
    "\n",
    "model = pickle.load(open(\"Otto_LightGBM_tfidf.pkl\", 'rb'))\n",
    "\n",
    "#输出每类的概率\n",
    "y_test_pred = model.predict_proba(X_test)\n",
    "print('finish')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(144368, 9)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_pred.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "#生成提交结果\n",
    "out_df = pd.DataFrame(y_test_pred)\n",
    "\n",
    "columns = np.empty(9, dtype=object)\n",
    "for i in range(9):\n",
    "    columns[i] = 'Class_' + str(i+1)\n",
    "\n",
    "out_df.columns = columns\n",
    "\n",
    "out_df = pd.concat([test_id,out_df], axis = 1)\n",
    "out_df.to_csv(\"LightGBM_tfidf.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原始特征和tfidf两种特征的LightGBM：0.44366（排名567位）  只有tfidf的lightGBM：0.46037 只有tfidf的测试效果没有两种特征一起用的效果好，但也不错"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
